search

restarone / violet_rails

an app engine for your business. Seamlessly implement business logic with a powerful API. Out of the box CMS, blog, forum and email functionality. Developer friendly & easily extendable for your next SaaS/XaaS project. Built with Rails 6, Devise, Sidekiq & PostgreSQL
https://violet.restarone.solutions
MIT License
96 stars 43 forks source link

fix flakiness in deploy action #338

Open donrestarone opened 2 years ago

donrestarone commented 2 years ago 
2022-03-28T23:22:18.4842000Z     ✔ 02 runner@localhost 80.758s
2022-03-28T23:22:18.4843946Z 01:32 deploy:assets:rsync
2022-03-28T23:22:18.4846336Z       01 rsync -av --delete  ./public/assets/ ubuntu@***:/var/www/violet/releases/20220328232047/public/assets/
2022-03-28T23:24:28.0841939Z       01 ssh: connect to host *** port 22: Connection timed out
2022-03-28T23:24:28.0857621Z       01 rsync: connection unexpectedly closed (0 bytes received so far) [sender]
2022-03-28T23:24:28.0860342Z       01 rsync error: unexplained error (code 255) at io.c(235) [sender=3.1.2]
2022-03-28T23:24:28.0865073Z #<Thread:0x00007f22141c3648@/home/runner/work/violet_rails/violet_rails/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.2/lib/sshkit/runners/parallel.rb:10 run> terminated with exception (report_on_exception is true):
2022-03-28T23:24:28.0872852Z /home/runner/work/violet_rails/violet_rails/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.2/lib/sshkit/runners/parallel.rb:15:in `rescue in block (2 levels) in execute': Exception while executing as ubuntu@***: rsync -av --delete  ./public/assets/ ubuntu@***:/var/www/violet/releases/20220328232047/public/assets/ exit status: 65280 (SSHKit::Runner::ExecuteError)
2022-03-28T23:24:28.0878036Z rsync -av --delete  ./public/assets/ ubuntu@***:/var/www/violet/releases/20220328232047/public/assets/ stdout: Nothing written
2022-03-28T23:24:28.0882484Z rsync -av --delete  ./public/assets/ ubuntu@***:/var/www/violet/releases/20220328232047/public/assets/ stderr: ssh: connect to host *** port 22: Connection timed out
2022-03-28T23:24:28.0885646Z rsync: connection unexpectedly closed (0 bytes received so far) [sender]
2022-03-28T23:24:28.0888286Z rsync error: unexplained error (code 255) at io.c(235) [sender=3.1.2]
2022-03-28T23:24:28.0891910Z    from /home/runner/work/violet_rails/violet_rails/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.2/lib/sshkit/runners/parallel.rb:11:in `block (2 levels) in execute'
2022-03-28T23:24:28.0898162Z /home/runner/work/violet_rails/violet_rails/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.2/lib/sshkit/command.rb:97:in `exit_status=': rsync -av --delete  ./public/assets/ ubuntu@***:/var/www/violet/releases/20220328232047/public/assets/ exit status: 65280 (SSHKit::Command::Failed)
2022-03-28T23:24:28.0902936Z rsync -av --delete  ./public/assets/ ubuntu@***:/var/www/violet/releases/20220328232047/public/assets/ stdout: Nothing written
2022-03-28T23:24:28.0907373Z rsync -av --delete  ./public/assets/ ubuntu@***:/var/www/violet/releases/20220328232047/public/assets/ stderr: ssh: connect to host *** port 22: Connection timed out
2022-03-28T23:24:28.0911016Z rsync: connection unexpectedly closed (0 bytes received so far) [sender]
2022-03-28T23:24:28.0913646Z rsync error: unexplained error (code 255) at io.c(235) [sender=3.1.2]
2022-03-28T23:24:28.0917267Z    from /home/runner/work/violet_rails/violet_rails/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.2/lib/sshkit/backends/local.rb:59:in `block in execute_command'
2022-03-28T23:24:28.0920376Z    from /opt/hostedtoolcache/Ruby/2.6.6/x64/lib/ruby/2.6.0/open3.rb:219:in `popen_run'
2022-03-28T23:24:28.0923035Z    from /opt/hostedtoolcache/Ruby/2.6.6/x64/lib/ruby/2.6.0/open3.rb:101:in `popen3'
2022-03-28T23:24:28.0926561Z    from /home/runner/work/violet_rails/violet_rails/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.2/lib/sshkit/backends/local.rb:44:in `execute_command'
2022-03-28T23:24:28.0930680Z    from /home/runner/work/violet_rails/violet_rails/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.2/lib/sshkit/backends/abstract.rb:148:in `block in create_command_and_execute'
2022-03-28T23:24:28.0934632Z    from /home/runner/work/violet_rails/violet_rails/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.2/lib/sshkit/backends/abstract.rb:148:in `tap'
2022-03-28T23:24:28.0938536Z    from /home/runner/work/violet_rails/violet_rails/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.2/lib/sshkit/backends/abstract.rb:148:in `create_command_and_execute'
2022-03-28T23:24:28.0942413Z    from /home/runner/work/violet_rails/violet_rails/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.2/lib/sshkit/backends/abstract.rb:80:in `execute'
2022-03-28T23:24:28.0944849Z    from /home/runner/work/violet_rails/violet_rails/vendor/bundle/ruby/2.6.0/gems/capistrano-local-precompile-1.2.0/lib/capistrano/local_precompile.rb:49:in `block (6 levels) in <top (required)>'
2022-03-28T23:24:28.0947286Z    from /home/runner/work/violet_rails/violet_rails/vendor/bundle/ruby/2.6.0/gems/capistrano-local-precompile-1.2.0/lib/capistrano/local_precompile.rb:45:in `each'
2022-03-28T23:24:28.0950044Z    from /home/runner/work/violet_rails/violet_rails/vendor/bundle/ruby/2.6.0/gems/capistrano-local-precompile-1.2.0/lib/capistrano/local_precompile.rb:45:in `block (5 levels) in <top (required)>'
2022-03-28T23:24:28.0952460Z    from /home/runner/work/violet_rails/violet_rails/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.2/lib/sshkit/backends/abstract.rb:31:in `instance_exec'
2022-03-28T23:24:28.0954635Z    from /home/runner/work/violet_rails/violet_rails/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.2/lib/sshkit/backends/abstract.rb:31:in `run'
2022-03-28T23:24:28.0956776Z    from /home/runner/work/violet_rails/violet_rails/vendor/bundle/ruby/2.6.0/gems/capistrano-3.16.0/lib/capistrano/dsl.rb:76:in `run_locally'
2022-03-28T23:24:28.0959214Z    from /home/runner/work/violet_rails/violet_rails/vendor/bundle/ruby/2.6.0/gems/capistrano-local-precompile-1.2.0/lib/capistrano/local_precompile.rb:38:in `block (4 levels) in <top (required)>'
2022-03-28T23:24:28.0962656Z    from /home/runner/work/violet_rails/violet_rails/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.2/lib/sshkit/backends/abstract.rb:31:in `instance_exec'
2022-03-28T23:24:28.0964976Z    from /home/runner/work/violet_rails/violet_rails/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.2/lib/sshkit/backends/abstract.rb:31:in `run'
2022-03-28T23:24:28.0968701Z    from /home/runner/work/violet_rails/violet_rails/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.2/lib/sshkit/runners/parallel.rb:12:in `block (2 levels) in execute'
2022-03-28T23:24:28.0972253Z (Backtrace restricted to imported tasks)
2022-03-28T23:24:28.0973341Z cap aborted!
2022-03-28T23:24:28.0975908Z SSHKit::Runner::ExecuteError: Exception while executing as ubuntu@***: rsync -av --delete  ./public/assets/ ubuntu@***:/var/www/violet/releases/20220328232047/public/assets/ exit status: 65280
2022-03-28T23:24:28.0978467Z rsync -av --delete  ./public/assets/ ubuntu@***:/var/www/violet/releases/20220328232047/public/assets/ stdout: Nothing written
2022-03-28T23:24:28.0981087Z rsync -av --delete  ./public/assets/ ubuntu@***:/var/www/violet/releases/20220328232047/public/assets/ stderr: ssh: connect to host *** port 22: Connection timed out
2022-03-28T23:24:28.0982940Z rsync: connection unexpectedly closed (0 bytes received so far) [sender]
2022-03-28T23:24:28.0984469Z rsync error: unexplained error (code 255) at io.c(235) [sender=3.1.2]
2022-03-28T23:24:28.0985265Z 
2022-03-28T23:24:28.0985291Z 
2022-03-28T23:24:28.0985726Z Caused by:
2022-03-28T23:24:28.0987656Z SSHKit::Command::Failed: rsync -av --delete  ./public/assets/ ubuntu@***:/var/www/violet/releases/20220328232047/public/assets/ exit status: 65280
2022-03-28T23:24:28.0990212Z rsync -av --delete  ./public/assets/ ubuntu@***:/var/www/violet/releases/20220328232047/public/assets/ stdout: Nothing written
2022-03-28T23:24:28.0992742Z rsync -av --delete  ./public/assets/ ubuntu@***:/var/www/violet/releases/20220328232047/public/assets/ stderr: ssh: connect to host *** port 22: Connection timed out
2022-03-28T23:24:28.0994577Z rsync: connection unexpectedly closed (0 bytes received so far) [sender]
2022-03-28T23:24:28.0996118Z rsync error: unexplained error (code 255) at io.c(235) [sender=3.1.2]
2022-03-28T23:24:28.0996923Z 
2022-03-28T23:24:28.0997463Z Tasks: TOP => deploy:assets:rsync
2022-03-28T23:24:28.0998694Z (See full trace by running task with --trace)
2022-03-28T23:24:28.1001108Z The deploy has failed with an error: Exception while executing as ubuntu@***: rsync -av --delete  ./public/assets/ ubuntu@***:/var/www/violet/releases/20220328232047/public/assets/ exit status: 65280
2022-03-28T23:24:28.1003635Z rsync -av --delete  ./public/assets/ ubuntu@***:/var/www/violet/releases/20220328232047/public/assets/ stdout: Nothing written
2022-03-28T23:24:28.1006355Z rsync -av --delete  ./public/assets/ ubuntu@***:/var/www/violet/releases/20220328232047/public/assets/ stderr: ssh: connect to host *** port 22: Connection timed out
2022-03-28T23:24:28.1008214Z rsync: connection unexpectedly closed (0 bytes received so far) [sender]
2022-03-28T23:24:28.1009845Z rsync error: unexplained error (code 255) at io.c(235) [sender=3.1.2]
2022-03-28T23:24:28.1010589Z 
2022-03-28T23:24:28.1010597Z 
2022-03-28T23:24:28.1011022Z ** DEPLOY FAILED
2022-03-28T23:24:28.1012166Z ** Refer to log/capistrano.log for details. Here are the last 20 lines:
2022-03-28T23:24:28.1012916Z 
2022-03-28T23:24:28.1012924Z 
2022-03-28T23:24:28.1013413Z  DEBUG [bf0524aa]    application (867 KiB)
2022-03-28T23:24:28.1013981Z 
2022-03-28T23:24:28.1014619Z  DEBUG [bf0524aa]        css/application-5bbc6a2f.css
2022-03-28T23:24:28.1015266Z 
2022-03-28T23:24:28.1015935Z  DEBUG [bf0524aa]        js/application-d6418d5c2576dcb74381.js
2022-03-28T23:24:28.1016628Z 
2022-03-28T23:24:28.1017102Z  DEBUG [bf0524aa]    trix (247 KiB)
2022-03-28T23:24:28.1017639Z 
2022-03-28T23:24:28.1018257Z  DEBUG [bf0524aa]        js/trix-be9c6c84b69252a44ed3.js
2022-03-28T23:24:28.1018914Z 
2022-03-28T23:24:28.1019327Z  DEBUG [bf0524aa]  
2022-03-28T23:24:28.1019801Z 
2022-03-28T23:24:28.1020211Z  DEBUG [bf0524aa]  
2022-03-28T23:24:28.1020694Z 
2022-03-28T23:24:28.1021300Z  DEBUG [bf0524aa]  WARNING in webpack performance recommendations: 
2022-03-28T23:24:28.1022020Z 
2022-03-28T23:24:28.1023000Z  DEBUG [bf0524aa]  You can limit the size of your bundles by using import() or require.ensure to lazy load some parts of your application.
2022-03-28T23:24:28.1024167Z 
2022-03-28T23:24:28.1025077Z  DEBUG [bf0524aa]  For more info visit https://webpack.js.org/guides/code-splitting/
2022-03-28T23:24:28.1025948Z 
2022-03-28T23:24:28.1027692Z  DEBUG [bf0524aa]  Child mini-css-extract-plugin node_modules/css-loader/dist/cjs.js??ref--5-1!node_modules/postcss-loader/s***/index.js??ref--5-2!node_modules/select2/dist/css/select2.css:
2022-03-28T23:24:28.1029139Z 
2022-03-28T23:24:28.1029887Z  DEBUG [bf0524aa]      Entrypoint mini-css-extract-plugin = *
2022-03-28T23:24:28.1030648Z 
2022-03-28T23:24:28.1031147Z  DEBUG [bf0524aa]         2 modules
2022-03-28T23:24:28.1031711Z 
2022-03-28T23:24:28.1032151Z  DEBUG [bf0524aa]  
2022-03-28T23:24:28.1032774Z 
2022-03-28T23:24:28.1033476Z   INFO [bf0524aa] Finished in 80.758 seconds with exit status 0 (successful).
2022-03-28T23:24:28.1034278Z 
2022-03-28T23:24:28.1035610Z   INFO [4267910e] Running rsync -av --delete  ./public/assets/ ubuntu@***:/var/www/violet/releases/20220328232047/public/assets/ as runner@localhost
2022-03-28T23:24:28.1036675Z 
2022-03-28T23:24:28.1037913Z  DEBUG [4267910e] Command: rsync -av --delete  ./public/assets/ ubuntu@***:/var/www/violet/releases/20220328232047/public/assets/
2022-03-28T23:24:28.1038899Z 
2022-03-28T23:24:28.1039630Z  DEBUG [4267910e]  ssh: connect to host *** port 22: Connection timed out
2022-03-28T23:24:28.1040424Z 
2022-03-28T23:24:28.1041227Z  DEBUG [4267910e]  rsync: connection unexpectedly closed (0 bytes received so far) [sender]
2022-03-28T23:24:28.1042100Z 
2022-03-28T23:24:28.1042873Z  DEBUG [4267910e]  rsync error: unexplained error (code 255) at io.c(235) [sender=3.1.2]
2022-03-28T23:24:28.1043728Z 
2022-03-28T23:24:28.1072534Z ##[error]Error: The process '/opt/hostedtoolcache/Ruby/2.6.6/x64/bin/bundle' failed with exit code 1
2022-03-28T23:24:28.1175525Z Post job cleanup.
2022-03-28T23:24:28.7995156Z The IP 52.251.59.48 is removed
2022-03-28T23:24:28.8162642Z Post job cleanup.
2022-03-28T23:24:28.9617069Z [command]/usr/bin/git version
2022-03-28T23:24:28.9738413Z git version 2.35.1
2022-03-28T23:24:28.9758911Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-03-28T23:24:28.9806533Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-03-28T23:24:29.0081741Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-03-28T23:24:29.0121337Z http.https://github.com/.extraheader
2022-03-28T23:24:29.0133133Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2022-03-28T23:24:29.0224162Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-03-28T23:24:29.0698294Z Cleaning up orphan processes
donrestarone commented 2 years ago

digging deeper into the error it looks like its a simple connection timeout. We should lengthen the wait time to the maximum (maybe 30 seconds?) 



Tasks: TOP => deploy:assets:rsync
(See full trace by running task with --trace)
The deploy has failed with an error: Exception while executing as ubuntu@***: rsync -av --delete  ./public/assets/ ubuntu@***:/var/www/violet/releases/20220413222925/public/assets/ exit status: 65280
rsync -av --delete  ./public/assets/ ubuntu@***:/var/www/violet/releases/20220413222925/public/assets/ stdout: Nothing written
rsync -av --delete  ./public/assets/ ubuntu@***:/var/www/violet/releases/20220413222925/public/assets/ stderr: ssh: connect to host *** port 22: Connection timed out
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(235) [sender=3.1.2]

end
donrestarone commented 2 years ago

possible solutions:

  1. https://stackoverflow.com/questions/11570209/capistrano-deploy-timeout
  2. https://stackoverflow.com/questions/12939027/rsync-over-ssh-timeout-in-ssh-or-rsync (already implemented)
donrestarone commented 2 years ago

new problem since 'fixing' (not yet confirmed) the rsync issue: looks like we need to set an SSH timeout on the target servers (client connection keep alive or something similar)

Screen Shot 2022-04-16 at 1 52 25 PM

still an issue (rsync isn't passing the additional parameter see deploy.rb):

Screen Shot 2022-04-16 at 2 02 52 PM
donrestarone commented 2 years ago

when set in deploy/environment.rb it worked!

2022-06-29T23:11:57.8727935Z     ✔ 02 runner@localhost 163.395s
2022-06-29T23:11:57.8728960Z 02:59 deploy:assets:rsync
2022-06-29T23:11:57.8730519Z       01 rsync -av --delete --timeout 300  ./public/assets/ ubuntu@***:/var/www/violet/releases/20220629230859/public/assets/
2022-06-29T23:11:59.4574684Z       01 sending incremental file list
2022-06-29T23:11:59.4745014Z       01 ./
donrestarone commented 2 years ago

its getting set to 300 didnt make the deployments more consistent -- reopening