Open azrle opened 5 years ago
Have you checked if chef exits on disconnect? If not, seems like retry could be problematic
In my case, it does not exist.
The run_command
tried to run chef
on a closed SSH session and generated Net::SSH::Disconnect
.
Hence, the chef
did not even start.
Checking Chef version...
Resolving cookbook dependencies...
Uploading the kitchen...
Generating solo config...
ERROR: Net::SSH::Disconnect: connection closed by remote hostRunning Chef: sudo chef-solo -c /var/tmp/chef-solo_xxx/solo.rb -j /var/tmp/chef-solo_xxx/dna.jso
n -W
However, risk exists as Net::SSH::Disconnect
may happen under other conditions.
That's surprising to see that part of the run take so long.
Anyway if it helps your situation I'd be fine with a configurable retry. But seems risky as a default.
It also surprises me too. I am going to conduct another investigation why it takes some much time. Maybe resolving dependencies part and rsync part.
Considering this error should not be a common case, and this fix is not a proper fix, I will try to add an option to support this workaround. Does it sound good to you?
Yeah I don't see an issue if it's configurable and off by default.
On Sun, Dec 23, 2018 at 23:40 Xuanzhong Wei notifications@github.com wrote:
It also surprises me too. I am going to conduct another investigation why it takes some much time. Maybe resolving dependencies part and rsync part.
Considering this error should not be a common case, and this fix is not a proper fix, I will try to add an option to support this workaround. Does it sound good to you?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/matschaffer/knife-solo/pull/536#issuecomment-449640634, or mute the thread https://github.com/notifications/unsubscribe-auth/AAACsjMd0VGgrzOs_9yVY0juTECYF07zks5u75XOgaJpZM4ZftGB .
--
-Mat
matschaffer.com
Hi @matschaffer,
I sometimes encounter
Net::SSH::Disconnect
when trying to runchef-solo
on slow machines.After some investigation, I found that the server indeed closed the connection since it has been idle for a while. The reason why the connection becomes idle is simply that the machine (
chef-solo
runs on) is slow and sometimes it takes time to do works. Hence, I try to set thessh_keepalive
andssh_keepalive_interval
to a reasonable value in order to send keepalive message when the connection is idle. However it does not work as expected.As the investigation goes, I found that
keepalive
andkeepalive_interval
only work for the case that you are running a slow command on the remote or you doNet::SSH::Session::loop
becausenet-ssh
is based on event and so is the keepalive ofnet-ssh
. I am not sure if there is any way to keep the connection alive and reuse the connection efficiently.However, it may be good to rescue from
Net::SSH::Disconnect
and retry the command. The caveat is that a command might be executed on the remote more than once.Not sure are these issues are related to this one. https://github.com/matschaffer/knife-solo/issues/272 https://github.com/matschaffer/knife-solo/issues/332