I'm experiencing intermittent SSH connection timeouts during the full kamal deploy process, specifically in the pull method. Interestingly, running kamal build pull separately works fine, but the same operation fails during the full deployment.
Environment
Kamal version: 1.7.1
Steps to Reproduce
Run kamal app remove && kamal deploy --verbose
Observe that the process fails during the pull method with a timeout error
Run kamal build pull separately
Observe that it works without issues
Error Message
DEBUG [4d2890b0] Running /usr/bin/env echo [2024-06-23T14:33:29Z] [myuser] Pulled image with version 3fbef4c6366422100f25a27d0376790eebd81461 >> .kamal/frontend_v2-audit.log on 5.161.193.74
DEBUG [4d2890b0] Command: /usr/bin/env echo [2024-06-23T14:33:29Z] [myuser] Pulled image with version 3fbef4c6366422100f25a27d0376790eebd81461 >> .kamal/frontend_v2-audit.log
Finished all in 231.3 seconds
ERROR (IO::TimeoutError): Exception while executing on host 5.161.193.74: Blocking operation timed out!
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/net-ssh-7.2.3/lib/net/ssh/buffered_io.rb:64:in `recv'
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/net-ssh-7.2.3/lib/net/ssh/buffered_io.rb:64:in `fill'
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/net-ssh-7.2.3/lib/net/ssh/connection/session.rb:275:in `block in ev_do_handle_events'
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/net-ssh-7.2.3/lib/net/ssh/connection/session.rb:271:in `each'
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/net-ssh-7.2.3/lib/net/ssh/connection/session.rb:271:in `ev_do_handle_events'
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/net-ssh-7.2.3/lib/net/ssh/connection/event_loop.rb:117:in `ev_select_and_postprocess'
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/net-ssh-7.2.3/lib/net/ssh/connection/event_loop.rb:30:in `process'
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/net-ssh-7.2.3/lib/net/ssh/connection/session.rb:226:in `process'
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/net-ssh-7.2.3/lib/net/ssh/connection/session.rb:179:in `block in loop'
<internal:kernel>:187:in `loop'
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/net-ssh-7.2.3/lib/net/ssh/connection/session.rb:179:in `loop'
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/sshkit-1.22.2/lib/sshkit/backends/netssh.rb:182:in `block in execute_command'
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/sshkit-1.22.2/lib/sshkit/backends/connection_pool.rb:65:in `with'
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/kamal-1.7.1/lib/kamal/sshkit_with_ext.rb:84:in `with_ssh'
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/sshkit-1.22.2/lib/sshkit/backends/netssh.rb:146:in `execute_command'
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/sshkit-1.22.2/lib/sshkit/backends/abstract.rb:148:in `block in create_command_and_execute'
<internal:kernel>:90:in `tap'
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/sshkit-1.22.2/lib/sshkit/backends/abstract.rb:148:in `create_command_and_execute'
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/sshkit-1.22.2/lib/sshkit/backends/abstract.rb:80:in `execute'
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/kamal-1.7.1/lib/kamal/cli/build.rb:61:in `block in pull'
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/sshkit-1.22.2/lib/sshkit/backends/abstract.rb:31:in `instance_exec'
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/sshkit-1.22.2/lib/sshkit/backends/abstract.rb:31:in `run'
/Users/myuser/.local/share/mise/installs/ruby/3.3.3/lib/ruby/gems/3.3.0/gems/kamal-1.7.1/lib/kamal/sshkit_with_ext.rb:117:in `block (2 levels) in execute'
Additional Information
The issue occurs consistently during full deployment but not when running kamal build pull separately
SSH agent is not running, but SSH_AUTH_SOCK is set
SSH configuration shows forward_agent: true
Questions
Why might the SSH connection work for kamal build pull but fail during kamal deploy?
Are there any known issues with SSH connections timing out during longer processes?
Could there be a problem with how the SSH connection is maintained or reused during the full deployment process?
Attempted Solutions
Increased SSH timeout to 300 seconds
Added more verbose logging and error handling
Verified SSH key presence and configuration
Any assistance in resolving this issue or suggestions for further debugging would be greatly appreciated.
I'm experiencing intermittent SSH connection timeouts during the full
kamal deployprocess, specifically in thepullmethod. Interestingly, runningkamal build pullseparately works fine, but the same operation fails during the full deployment.Environment
Steps to Reproduce
kamal app remove && kamal deploy --verbosepullmethod with a timeout errorkamal build pullseparatelyError Message
Additional Information
kamal build pullseparatelyforward_agent: trueQuestions
kamal build pullbut fail duringkamal deploy?Attempted Solutions
Any assistance in resolving this issue or suggestions for further debugging would be greatly appreciated.