SSH disconnects when the laptop goes to sleep even for a short time while the native terminal is running

[thechargedneutron]

• VSCode Version: 1.62.0-insider
• Local OS Version: macOS Big Sur 11.5.2
• Remote OS Version: Ubuntu 18.04 LTS
• Remote Extension/Connection Type: SSH
• Logs:

Steps to Reproduce:

1. Open native terminal in macOS and connect to a remote server. Connect to the same server with vscode.
2. Close the laptop lid and instantly open it up. Alternatively, let the laptop sleep due to inactivity and then instantly wake it up using the trackpad.

The vscode is now disconnected and asks me to reload window. The macOS terminal is still connected to the remote and works perfectly. Is there a way to delay the session time-out time in vscode? It is a bit annoying to reload every time the screen goes off for a minute. I have to increase the sleep duration to tackle this issue.

Does this issue occur when you try this locally?: Yes Does this issue occur when you try this locally and all extensions are disabled?: Yes

[tanhakabir]

@roblourens I'm guessing this has to do with the ssh process closing when the computer goes to sleep and when we try to reconnect, we aren't able to within the 5 times so we close the connection?

[roblourens]

It should be trying to reconnect at this point. Please share the log from the Remote-SSH output channel, it should tell us why it failed to reconnect

[thechargedneutron]

@roblourens After I put the laptop lid down and instantly put it up, the password prompt comes, if I do not enter the password and click exit I get the following log. Though, my concern is that vscode should not ask for password again if I instantly open up my lid again (similar to macOS native terminal).

[19:57:04.914] [Forwarding server 52217] Got connection 3
[19:57:04.925] ------

[19:57:04.925] SSH Resolver called for "ssh-remote+`hostname`", attempt 2, (Reconnection)
[19:57:04.925] SSH Resolver called for host: `hostname`
[19:57:04.925] Setting up SSH remote `"hostname"`
[19:57:04.927] Acquiring local install lock: /var/folders/bn/wchw2lf52zdc_y9p6bm7217m0000gn/T/vscode-remote-ssh-76b40f39-install.lock
[19:57:04.934] Looking for existing server data file at /Users/thechargedneutron/Library/Application Support/Code - Insiders/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-76b40f39-34eeebdd66033633ac22efd5fab928ae37916168-0.65.8/data.json
[19:57:04.938] Found local server running: {"remoteListeningOn":{"port":37801},"osReleaseId":"ubuntu","arch":"x86_64","webUiAccessToken":"","sshAuthSock":"","display":"","tmpDir":"/run/user/41042","platform":"linux","connectionToken":"a11a111a-1a11-11aa-1a1a-11a11aa111aa","pid":83938,"ipcHandlePath":"/var/folders/bn/wchw2lf52zdc_y9p6bm7217m0000gn/T/vscode-ssh-askpass-2ef0d9ce17bbe074fbc7bf86195d17bc1a93c36e.sock","socksPort":52193,"startupTime":1635555338990}
[19:57:04.938] Running server is stale. Ignoring
[19:57:04.939] Using commit id "34eeebdd66033633ac22efd5fab928ae37916168" and quality "insider" for server
[19:57:04.942] Install and start server if needed
[19:57:04.954] PATH: /Users/thechargedneutron/.rbenv/shims:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin:/Users/thechargedneutron/Library/Android/sdk/emulator:/Users/thechargedneutron/Library/Android/sdk/tools:/Users/thechargedneutron/Library/Android/sdk/tools/bin:/Users/thechargedneutron/Library/Android/sdk/platform-tools
[19:57:04.954] Checking ssh with "ssh -V"
[19:57:04.963] > OpenSSH_8.1p1, LibreSSL 2.7.3

[19:57:04.964] askpass server listening on /var/folders/bn/wchw2lf52zdc_y9p6bm7217m0000gn/T/vscode-ssh-askpass-243fffb2716a3c88297ba544bff67b3ee7beb0c2.sock
[19:57:04.964] Spawning local server with {"serverId":1,"ipcHandlePath":"/var/folders/bn/wchw2lf52zdc_y9p6bm7217m0000gn/T/vscode-ssh-askpass-0dabf9d3e7ca1a73ca37809a7142fdfd3c525eb0.sock","sshCommand":"ssh","sshArgs":["-v","-T","-D","52243","-o","ConnectTimeout=15","`hostname`"],"dataFilePath":"/Users/thechargedneutron/Library/Application Support/Code - Insiders/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-76b40f39-34eeebdd66033633ac22efd5fab928ae37916168-0.65.8/data.json"}
[19:57:04.964] Local server env: {"DISPLAY":"1","ELECTRON_RUN_AS_NODE":"1","SSH_ASKPASS":"/Users/thechargedneutron/.vscode-insiders/extensions/ms-vscode-remote.remote-ssh-0.65.8/out/local-server/askpass.sh","VSCODE_SSH_ASKPASS_NODE":"/Applications/Visual Studio Code - Insiders.app/Contents/Frameworks/Code - Insiders Helper (Renderer).app/Contents/MacOS/Code - Insiders Helper (Renderer)","VSCODE_SSH_ASKPASS_MAIN":"/Users/thechargedneutron/.vscode-insiders/extensions/ms-vscode-remote.remote-ssh-0.65.8/out/askpass-main.js","VSCODE_SSH_ASKPASS_HANDLE":"/var/folders/bn/wchw2lf52zdc_y9p6bm7217m0000gn/T/vscode-ssh-askpass-243fffb2716a3c88297ba544bff67b3ee7beb0c2.sock"}
[19:57:04.967] Spawned 83978
[19:57:05.101] > local-server-1> Spawned ssh, pid=83983
[19:57:05.102] stderr> OpenSSH_8.1p1, LibreSSL 2.7.3
[19:57:05.324] stderr> debug1: Server host key: ecdsa-sha2-nistp256 SHA256:n09TbVQEP2lfMtVM1hdZAsZs5lfm4IakAEmuWgAcfOg
[19:57:05.620] Got askpass request: {"request":"Enter passphrase for key '/Users/thechargedneutron/.ssh/id_rsa':"}
[19:57:05.621] Detected passphrase message
[19:57:05.621] Listening for interwindow password on /var/folders/bn/wchw2lf52zdc_y9p6bm7217m0000gn/T/vscode-ssh-askpass-edfb48cf23ea357f834d86795ca6b6fb713bbc2c.sock
[19:57:05.621] Writing password prompt to globalState
[19:57:10.883] Passphrase dialog canceled
[19:57:10.883] Interactor signaled cancel
[19:57:10.884] Cleaning up other-window auth server
[19:57:10.903] stderr> `username@hostname`: Permission denied (publickey).
[19:57:10.904] > local-server-1> ssh child died, shutting down
[19:57:10.906] Local server exit: 0
[19:57:10.906] Received install output: local-server-1> Spawned ssh, pid=83983
OpenSSH_8.1p1, LibreSSL 2.7.3
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:n09TbVQEP2lfMtVM1hdZAsZs5lfm4IakAEmuWgAcfOg
`username@hostname`: Permission denied (publickey).
local-server-1> ssh child died, shutting down

[19:57:10.908] Resolver error: Error: Permission denied (publickey).
    at Function.Create (/Users/thechargedneutron/.vscode-insiders/extensions/ms-vscode-remote.remote-ssh-0.65.8/out/extension.js:1:397041)
    at /Users/thechargedneutron/.vscode-insiders/extensions/ms-vscode-remote.remote-ssh-0.65.8/out/extension.js:1:395235
    at Object.t.handleInstallOutput (/Users/thechargedneutron/.vscode-insiders/extensions/ms-vscode-remote.remote-ssh-0.65.8/out/extension.js:1:395620)
    at Object.e [as tryInstallWithLocalServer] (/Users/thechargedneutron/.vscode-insiders/extensions/ms-vscode-remote.remote-ssh-0.65.8/out/extension.js:1:435292)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async /Users/thechargedneutron/.vscode-insiders/extensions/ms-vscode-remote.remote-ssh-0.65.8/out/extension.js:1:452665
    at async Object.t.withShowDetailsEvent (/Users/thechargedneutron/.vscode-insiders/extensions/ms-vscode-remote.remote-ssh-0.65.8/out/extension.js:1:456460)
    at async /Users/thechargedneutron/.vscode-insiders/extensions/ms-vscode-remote.remote-ssh-0.65.8/out/extension.js:1:433816
    at async C (/Users/thechargedneutron/.vscode-insiders/extensions/ms-vscode-remote.remote-ssh-0.65.8/out/extension.js:1:430392)
    at async Object.t.resolveWithLocalServer (/Users/thechargedneutron/.vscode-insiders/extensions/ms-vscode-remote.remote-ssh-0.65.8/out/extension.js:1:433431)
    at async Object.t.resolve (/Users/thechargedneutron/.vscode-insiders/extensions/ms-vscode-remote.remote-ssh-0.65.8/out/extension.js:1:454065)
    at async /Users/thechargedneutron/.vscode-insiders/extensions/ms-vscode-remote.remote-ssh-0.65.8/out/extension.js:1:526942
[19:57:10.910] ------

[roblourens]

Well, the socket might close for some reason in the time that the laptop was asleep. Seems that sometimes ssh in the terminal will survive a brief sleep, sometimes not, same for vscode. Also, we implement our own timeout where once the remote is not reachable for some amount of time, we assume it's disconnected and try to reconnect. So I have to just call this "by design". @alexdima I wonder whether it could be improved for scenarios like this, if we can check the previous socket again after waking from sleep.

It's much more smooth if you set up SSH with your passphrase in the ssh agent so you don't need to type anything to connect

[alexdima]

@roblourens The core will invoke the resolver again. You can definitely return the same port again from the resolver if the previous SSH tunnel is still healthy. For example, once it started the server, Remote WSL always returns the same port from the resolver.

[thechargedneutron]

Honestly, I think there should be at least this level of robustness to keep the session alive for cases when the laptop has just gone to sleep. If I set my screen sleep time to 10 minutes the ssh remains active for this long. But if I set my screen sleep to 1 minute the ssh disconnects as soon as the screen goes off even for a few seconds. I appreciate the your efforts in discussing this issue. Thanks!

[roblourens]

We have the ability to test the connection again, but wouldn't it make more sense for the core to do that?

[alexdima]

We have the ability to test the connection again, but wouldn't it make more sense for the core to do that?

Doing that in all cases means that in many cases we would waste around 30s (the timeout we have) trying to connect to a possibly broken resolver port. So we always just invoke the resolver which can trivially return the same port if it so wishes.

e.g. of what you could do in the SSH resolver (if you want to):

class Resolver {
    private lastPort: number = 0;
    private lastPortReusedTime: number = 0;

    public resolve(ctx) {
        const shouldReuseLastPort = (
            this.lastPort
            && (Date.now() - this.lastPortReusedTime > 5 * 60 * 1000) // the port was not reused within the past 5 minutes
        );
        if (shouldReuseLastPort /* && tunnel process still alive */) {
            // try to reuse the same tunnel, maybe it still works fine
            this.lastPortReusedTime = Date.now();
            return this.lastPort;
        }
        this.lastPortReusedTime = 0;
        // ...
        // this.lastPort = ...
        return this.lastPort;
    }
}

The core calling the resolver gives the most freedom. Resolvers that know that they are really disconnected can recreate the connection, while resolvers which don't know can try to be lucky and return first the old port and only on a subsequent resolve call create a new tunnel.

What do you think?

[roblourens]

If we reuse the port, and it turns out to be stuck, will vscode spend 30s trying to connect to it? Because that will happen sometimes. The resolver can also ping the server through the port with a shorter timeout.

[alexdima]

If we reuse the port, and it turns out to be stuck, will vscode spend 30s trying to connect to it? Because that will happen sometimes. The resolver can also ping the server through the port with a shorter timeout.

:+1: Yes, that is exactly the point I was trying to make. When the core tries to reconnect to a host:port provided by the resolver, it sets a 30s timeout. If the timeout fires without the connection succeeding, it goes again to the resolver to resolve again and so on...

[roblourens]

@tanhakabir I've been wanting to do an optimization like this. We can ping the /delay-shutdown endpoint on the previous resolved port to check whether it's still responding, similar to what the install script does. If it responds, we can reuse the port. But if the resolver is invoked again within a short amount of time we would assume it's not healthy and open a new connection.

[jamestut]

OK so I have the same symptoms, but a different configuration. In a nutshell, my configuration is like:

SSH on mac host to 127.0.0.1 -> Unix socket -> VSOCK -> Linux' guest SSH.

• VSCode Version: 1.61
• Local OS Version: macOS 11.6.1
• Remote OS Version: Fedora 35

As the connection is performed over VSOCK, it is guaranteed to be stable (no disconnects). I have tested this using both SSH, netcat, and socat on terminal, then put my MacBook to sleep, and all of them are stills connected upon waking up the MacBook. However, VSCode immediately ask for reloading the window.

One thing to note is that the time on the VM momentarily drifts backwards for about 5 seconds before it syncs back to the host time. Also for what its worth, this is a custom hypervisor based on macOS Hypervisor.framework.

[cpbotha]

VSCode 1.69.2 on macOS 12.4 (M1 MBA over wifi, FWIW) connecting via remote SSH to Linux.

Close and open lid immediately, VSCode reports "Cannot reconnect. Please reload the window." -- jupyter notebook state is lost (any running processes gone), terminal state is lost.

Straight iTerm remains connected throughout all of this, I can continue typing my CLI invocation across resumes.

In my ideal world, I would be able to close laptop, move to a different location, and open to continue my work without losing state. :)

[dinsausti]

I have this issue on VSCode running on Windows10 with a remote SSH connection to an EC2 instance. My running Jupyter notebooks halt on sleep and I lose my progress. I too would like to figure out a solution for this as I don't have the ability to turn sleep off.

[jamestut]

Hi all, the problem on macOS is that Chrome is the one to blame: Chrome will disconnect active WebSocket connection, even the one made locally to 127.0.0.1 (e.g. the extension's internal proxy), when it detects that the mac is put to sleep. We can simulate trigger the system sleep right away on macOS by using the pmset sleepnow command (this will also instantly trigger Chrome to drop its connection). Any TCP connections from other apps including SSH to localhost should continue to work and not dropped, but that's not the case with Chrome.

Also FWIW, Safari does not do this.

Not sure about Windows as I don't have a Windows machine for the time being, but I think on Windows it is not related to the Chrome's problem I mentioned AFAIK.

[jamestut]

Hi @kieferrm @eleanorjboyd @roblourens, apologise beforehand for tagging. However, considering that this is Google Chrome's issue of cutting off WebSocket connection when the laptop is put to standby, I wonder what are your approach in tackling this problem?

Are you going to patch the underlying Electron code on the upcoming VSCode to fix this behaviour, or are you going to make this extension be able to persist states between reconnection?

[roblourens]

I wouldn't mess with that, we just need to make sure that we are reconnecting cleanly when waking up again. We haven't had time to look into this though. Thanks for the tip about Chrome though!

[movy]

Not exactly related to disconnect-on-sleep issue, but just an UX improvement idea: currently VSCode allows at most 8 reconnect attempts before it gives up and requires window reload, is it possible to set max to 20 attempts for example? Disconnections can be hard to avoid for people on the move, and when it comes to reconnection I observe vastly different behaviour between VPS providers, some of them allow faster reconnection while with others it takes forever, especially when VPN is involved. I guess it all boils down to their routers config and is beyond our control, but to make VSCode more resilient, probably trying reconnecting for a bit longer is sufficient in such scenarios.

[eleanorjboyd]

Hi @movy, happy to talk more in depth on this if you create another issue but if you set the max reconnection attempts to null it actually tries many more times than 8 (somewhere in the hundreds). We should update that setting to reflect that but as a fix for now to get more attempts just set that value to null- issue to track that is here: https://github.com/microsoft/vscode-remote-release/issues/7552. Thanks

[shrijayan]

OK so I have the same symptoms, but a different configuration. In a nutshell, my configuration is like:

SSH on mac host to 127.0.0.1 -> Unix socket -> VSOCK -> Linux' guest SSH.

• VSCode Version: 1.61
• Local OS Version: macOS 11.6.1
• Remote OS Version: Fedora 35

As the connection is performed over VSOCK, it is guaranteed to be stable (no disconnects). I have tested this using both SSH, netcat, and socat on terminal, then put my MacBook to sleep, and all of them are stills connected upon waking up the MacBook. However, VSCode immediately ask for reloading the window.

One thing to note is that the time on the VM momentarily drifts backwards for about 5 seconds before it syncs back to the host time. Also for what its worth, this is a custom hypervisor based on macOS Hypervisor.framework.

How to do this any step-by-step guide. I am trying to connect SSH server using VS Code but when my laptop goes to sleep the process that runs stops and the server disconnects again I need to start from first.