Extensions keeping the extension host busy lead to disconnection

[mykter]

• VSCode Version: Insiders 2019-07-31
• Local OS Version: Win 7
• Remote OS Version: Ubuntu 18.04 (on VMWare, running locally)
• Remote Extension/Connection Type: SSH

Steps to Reproduce:

1. Connect to a remote host
2. Do some work for a while (or don't!)

This is my variant of microsoft/vscode-remote-release#246.

The remote host in this case is a VM on the same machine, so the network connection is fast and reliable. I frequently get the popup saying remote host has disconnected, reconnecting in 5 seconds. It duly reconnects.

I can dismiss the popup, and carry on working, doing tasks that definitely require talking to the remote host. So it appears that it thinks it's been disconnected, when the connection is still very much alive and well.

It's not consistent, and I haven't worked out a pattern of when it happens. It's definitely bursty though - if I get one disconnection there will often be another a few seconds later.

There are no logs from any source at Trace level that correspond with the "you've been disconnected" popup appearing. There are plenty of logs that appear when it actually establishes a new connection. You can see all the old sshd instances still running on the remote host.

[roblourens]

Thanks for forking this and for apparently putting up with this issue for the past 2.5 months!

It sounds like you have looked at these and nothing really stands out, but to be complete, would you share

• The Remote-SSH output channel log
• The console output from the developer tools (under Help menu)

I am kind of stumped. Maybe we need more logging around disconnection in general. @alexandrudima tl;dr, I think the weird part is

I can dismiss the popup, and carry on working, doing tasks that definitely require talking to the remote host. So it appears that it thinks it's been disconnected, when the connection is still very much alive and well.

and I don't know why we would be able to send and receive messages with the remote EH while thinking that we are still disconnected.

[mykter]

Thanks for forking this and for apparently putting up with this issue for the past 2.5 months!

You're welcome, the improved experience of working with vscode on my host instead of my slow VM is worth hitting escape from time to time!

Remote - SSH output, this is after an initial connection and 3 reconnects

``` remote-ssh@0.44.2 win32 x64 SSH Resolver called for "ssh-remote+pokom3vm", attempt 1 SSH Resolver called for host: pokom3vm Setting up SSH remote "pokom3vm" Using commit id "92da68a71cfb60bd3b9b0d7fbfb2a7e1fff9dbaf" and quality "insider" for server Testing ssh with ssh -V ssh exited with code: 0 Got stderr from ssh: OpenSSH_7.1p2, OpenSSL 1.0.2g 1 Mar 2016 Running script with connection command: "ssh" -o ClearAllForwardings=true "pokom3vm" bash Install and start server if needed > > Running remote connection script > > Found old VS Code install fd5985489982fe14381dd56e6f0b29c53d5f991b, attempting t > o clean up > Deleting old install from /home/mmacnair/.vscode-server-insiders/bin/fd598548998 > 2fe14381dd56e6f0b29c53d5f991b > > Installing to /home/mmacnair/.vscode-server-insiders/bin/92da68a71cfb60bd3b9b0d7 > fbfb2a7e1fff9dbaf... > Downloading with wget > > Download complete > > SSH_CONNECTION=192.168.238.2 49841 192.168.238.139 22 > LANG=en_GB.UTF-8 > XDG_SESSION_ID=3 > USER=mmacnair > PWD=/home/mmacnair > HOME=/home/mmacnair > SSH_CLIENT=192.168.238.2 49841 22 > MAIL=/var/mail/mmacnair > SHELL=/bin/bash > SHLVL=2 > VSCODE_AGENT_FOLDER=/home/mmacnair/.vscode-server-insiders > LOGNAME=mmacnair > DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus > XDG_RUNTIME_DIR=/run/user/1000 > PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/us > r/local/games > _=/usr/bin/printenv > OLDPWD=/home/mmacnair/.vscode-server-insiders/bin/92da68a71cfb60bd3b9b0d7fbfb2a7 > e1fff9dbaf > Starting server... > cat: /home/mmacnair/.vscode-server-insiders/.92da68a71cfb60bd3b9b0d7fbfb2a7e1fff > 9dbaf.log: No such file or directory > Waiting for server log... > > > * > * Reminder: You may only use this software with Visual Studio family products, > * as described in the license (https://go.microsoft.com/fwlink/?linkid=2077057) > * > > a00ab847-3c24-43f3-b360-2cce5bf7d169==34175==45481== > "install" terminal command done Received install output: a00ab847-3c24-43f3-b360-2cce5bf7d169==34175==45481== Server is listening on port 34175 Running script with connection command: "ssh" -L 127.0.0.1:34175:127.0.0.1:34175 "pokom3vm" bash Spawning tunnel with: type "C:\Users\mmacnair\AppData\Local\Temp\vscode-linux-multi-line-command-pokom3vm-102433935.sh" | "ssh" -L 127.0.0.1:34175:127.0.0.1:34175 "pokom3vm" bash > > Connected to SSH Host - Please do not close this terminal > Spawned SSH tunnel between local port 34175 and remote port 34175 Waiting for ssh tunnel to be ready Tunneling remote port 34175 to local port 34175 Resolved "ssh-remote+pokom3vm" to "127.0.0.1:34175" ------ Log Level: Trace SSH Resolver called for "ssh-remote+pokom3vm", attempt 2 SSH Resolver called for host: pokom3vm Setting up SSH remote "pokom3vm" Using commit id "92da68a71cfb60bd3b9b0d7fbfb2a7e1fff9dbaf" and quality "insider" for server Running script with connection command: "ssh" -o ClearAllForwardings=true "pokom3vm" bash Install and start server if needed "install" terminal received data: "" "install" terminal received data: "Running remote connection script[?25l [?25h" "install" terminal received data: "Found existing installation at /home/mmacnair/.vscode-server-insiders/bin/92da68[?25l a71cfb60bd3b9b0d7fbfb2a7e1fff9dbaf... Found running server...  * * Reminder: You may only use this software with Visual Studio family products, * as described in the license (https://go.microsoft.com/fwlink/?linkid=2077057) *  Checking server status with wget 99a99512-8f1c-439b-b4ce-c32ac993f49d==34175==45481== [?25h" "install" terminal command done Received install output: 99a99512-8f1c-439b-b4ce-c32ac993f49d==34175==45481== Server is listening on port 34175 Running script with connection command: "ssh" -L 127.0.0.1:10173:127.0.0.1:34175 "pokom3vm" bash Spawning tunnel with: type "C:\Users\mmacnair\AppData\Local\Temp\vscode-linux-multi-line-command-pokom3vm-377425162.sh" | "ssh" -L 127.0.0.1:10173:127.0.0.1:34175 "pokom3vm" bash "SSH Tunnel" terminal received data: "" "SSH Tunnel" terminal received data: "Connected to SSH Host - Please do not close this terminal[?25l [?25h" Spawned SSH tunnel between local port 10173 and remote port 34175 Waiting for ssh tunnel to be ready Tunneling remote port 34175 to local port 10173 Resolved "ssh-remote+pokom3vm" to "127.0.0.1:10173" ------ SSH Resolver called for "ssh-remote+pokom3vm", attempt 3 SSH Resolver called for host: pokom3vm Setting up SSH remote "pokom3vm" Using commit id "92da68a71cfb60bd3b9b0d7fbfb2a7e1fff9dbaf" and quality "insider" for server Running script with connection command: "ssh" -o ClearAllForwardings=true "pokom3vm" bash Install and start server if needed "install" terminal received data: "" "install" terminal received data: "Running remote connection script[?25l [?25h" "install" terminal received data: "Found existing installation at /home/mmacnair/.vscode-server-insiders/bin/92da68[?25l a71cfb60bd3b9b0d7fbfb2a7e1fff9dbaf... Found running server...  * * Reminder: You may only use this software with Visual Studio family products, * as described in the license (https://go.microsoft.com/fwlink/?linkid=2077057) *  Checking server status with wget 1fcb6fa5-8d06-4ec1-8c5e-84262dc2ff67==34175==45481== [?25h" "install" terminal command done Received install output: 1fcb6fa5-8d06-4ec1-8c5e-84262dc2ff67==34175==45481== Server is listening on port 34175 Running script with connection command: "ssh" -L 127.0.0.1:1832:127.0.0.1:34175 "pokom3vm" bash Spawning tunnel with: type "C:\Users\mmacnair\AppData\Local\Temp\vscode-linux-multi-line-command-pokom3vm-990782777.sh" | "ssh" -L 127.0.0.1:1832:127.0.0.1:34175 "pokom3vm" bash "SSH Tunnel" terminal received data: "" "SSH Tunnel" terminal received data: "Connected to SSH Host - Please do not close this terminal[?25l [?25h" Spawned SSH tunnel between local port 1832 and remote port 34175 Waiting for ssh tunnel to be ready Tunneling remote port 34175 to local port 1832 Resolved "ssh-remote+pokom3vm" to "127.0.0.1:1832" ------ SSH Resolver called for "ssh-remote+pokom3vm", attempt 4 SSH Resolver called for host: pokom3vm Setting up SSH remote "pokom3vm" Using commit id "92da68a71cfb60bd3b9b0d7fbfb2a7e1fff9dbaf" and quality "insider" for server Running script with connection command: "ssh" -o ClearAllForwardings=true "pokom3vm" bash Install and start server if needed "install" terminal received data: "" "install" terminal received data: "Running remote connection script[?25l [?25h" "install" terminal received data: "Found existing installation at /home/mmacnair/.vscode-server-insiders/bin/92da68[?25l a71cfb60bd3b9b0d7fbfb2a7e1fff9dbaf... Found running server...[?25h" "install" terminal received data: "[?25l  * * Reminder: You may only use this software with Visual Studio family products, * as described in the license (https://go.microsoft.com/fwlink/?linkid=2077057) *  Checking server status with wget 8b7f04e5-eb82-485e-b070-c545ff034667==34175==45481== [?25h" "install" terminal command done Received install output: 8b7f04e5-eb82-485e-b070-c545ff034667==34175==45481== Server is listening on port 34175 Running script with connection command: "ssh" -L 127.0.0.1:24712:127.0.0.1:34175 "pokom3vm" bash Spawning tunnel with: type "C:\Users\mmacnair\AppData\Local\Temp\vscode-linux-multi-line-command-pokom3vm-654528052.sh" | "ssh" -L 127.0.0.1:24712:127.0.0.1:34175 "pokom3vm" bash "SSH Tunnel" terminal received data: "" "SSH Tunnel" terminal received data: "Connected to SSH Host - Please do not close this terminal[?25l [?25h" Spawned SSH tunnel between local port 24712 and remote port 34175 Waiting for ssh tunnel to be ready Tunneling remote port 34175 to local port 24712 Resolved "ssh-remote+pokom3vm" to "127.0.0.1:24712" ------ ```

Finally caught the Window output and devtools console output when the disconnect happens. Same behaviour as in microsoft/vscode-remote-release#264 (but I forgot to copy it in when raising this issue).

The console doesn't have timestamps, but this output appeared after the dialogue had timed out, nothing was logged when the disconnection dialogue appeared.

Devtools console output

``` TRACE load() - enter m {scheme: "vscode-remote", authority: "ssh-remote+pokom3vm", path: "/home/mmacnair/sdl/Dockerfile", query: "", fragment: "", …} workbench.main.js:1449 TRACE terminalInstance#ctor (id: 18) {name: "install", executable: "cmd.exe", args: "/c (type "C:\Users\mmacnair\AppData\Local\Temp\vsc…lForwardings=true "pokom3vm" bash) & exit /b 0", cwd: m, waitOnExit: undefined, …} workbench.main.js:1449 TRACE IPty#spawn cmd.exe /c (type "C:\Users\mmacnair\AppData\Local\Temp\vscode-linux-multi-line-command-pokom3vm-964987859.sh" | "ssh" -o ClearAllForwardings=true "pokom3vm" bash) & exit /b 0 {name: "cmd.exe", cwd: "c:\Users\mmacnair", env: {…}, cols: undefined, rows: undefined, …} workbench.main.js:1449 DEBUG Terminal process ready (shellProcessId: 10640) workbench.main.js:1449 TRACE IPty#kill workbench.main.js:1449 DEBUG Terminal process exit (id: 18) with code 0 workbench.main.js:1449 DEBUG Terminal process exit (id: 18) state 5 workbench.main.js:1449 TRACE terminalInstance#dispose (id: 18) workbench.main.js:1449 TRACE terminalInstance#ctor (id: 19) {name: "SSH Tunnel", executable: "cmd.exe", args: "/c (type "C:\Users\mmacnair\AppData\Local\Temp\vsc…:127.0.0.1:32921 "pokom3vm" bash) & exit /b 0", cwd: m, waitOnExit: undefined, …} workbench.main.js:1449 TRACE IPty#spawn cmd.exe /c (type "C:\Users\mmacnair\AppData\Local\Temp\vscode-linux-multi-line-command-pokom3vm-381622846.sh" | "ssh" -L 127.0.0.1:53641:127.0.0.1:32921 "pokom3vm" bash) & exit /b 0 {name: "cmd.exe", cwd: "c:\Users\mmacnair", env: {…}, cols: undefined, rows: undefined, …} workbench.main.js:1449 DEBUG Terminal process ready (shellProcessId: 10172) ```

Window output

``` [2019-08-06 12:25:58.755] [renderer1] [trace] terminalInstance#ctor (id: 18) {"name":"install","executable":"cmd.exe","args":"/c (type \"C:\\Users\\mmacnair\\AppData\\Local\\Temp\\vscode-linux-multi-line-command-pokom3vm-964987859.sh\" | \"ssh\" -o ClearAllForwardings=true \"pokom3vm\" bash) & exit /b 0","cwd":{"$mid":1,"path":"/C:/Users/mmacnair","scheme":"file"},"ignoreConfigurationCwd":true,"hideFromUser":true} [2019-08-06 12:25:58.759] [renderer1] [trace] IPty#spawn cmd.exe /c (type "C:\Users\mmacnair\AppData\Local\Temp\vscode-linux-multi-line-command-pokom3vm-964987859.sh" | "ssh" -o ClearAllForwardings=true "pokom3vm" bash) & exit /b 0 {"name":"cmd.exe","cwd":"c:\\Users\\mmacnair","env":{...,"TERM_PROGRAM":"vscode","TERM_PROGRAM_VERSION":"1.37.0-insider","LANG":"en_US.UTF-8","COLORTERM":"truecolor"},"experimentalUseConpty":false,"conptyInheritCursor":true} [2019-08-06 12:25:59.289] [renderer1] [debug] Terminal process ready (shellProcessId: 10640) [2019-08-06 12:26:00.380] [renderer1] [trace] IPty#kill [2019-08-06 12:26:00.381] [renderer1] [debug] Terminal process exit (id: 18) with code 0 [2019-08-06 12:26:00.382] [renderer1] [debug] Terminal process exit (id: 18) state 5 [2019-08-06 12:26:00.383] [renderer1] [trace] terminalInstance#dispose (id: 18) [2019-08-06 12:26:01.394] [renderer1] [trace] terminalInstance#ctor (id: 19) {"name":"SSH Tunnel","executable":"cmd.exe","args":"/c (type \"C:\\Users\\mmacnair\\AppData\\Local\\Temp\\vscode-linux-multi-line-command-pokom3vm-381622846.sh\" | \"ssh\" -L 127.0.0.1:53641:127.0.0.1:32921 \"pokom3vm\" bash) & exit /b 0","cwd":{"$mid":1,"path":"/C:/Users/mmacnair","scheme":"file"},"ignoreConfigurationCwd":true,"hideFromUser":true} [2019-08-06 12:26:01.398] [renderer1] [trace] IPty#spawn cmd.exe /c (type "C:\Users\mmacnair\AppData\Local\Temp\vscode-linux-multi-line-command-pokom3vm-381622846.sh" | "ssh" -L 127.0.0.1:53641:127.0.0.1:32921 "pokom3vm" bash) & exit /b 0 {"name":"cmd.exe","cwd":"c:\\Users\\mmacnair","env":{...,"TERM_PROGRAM":"vscode","TERM_PROGRAM_VERSION":"1.37.0-insider","LANG":"en_US.UTF-8","COLORTERM":"truecolor"},"experimentalUseConpty":false,"conptyInheritCursor":true} [2019-08-06 12:26:01.930] [renderer1] [debug] Terminal process ready (shellProcessId: 10172) ```

This looks interesting, but given the timing is wrong I wonder if it's the installer killing off the old instance that's still running, rather than the actual cause of the "disconnect"?

workbench.main.js:1449 TRACE IPty#kill
workbench.main.js:1449 DEBUG Terminal process exit (id: 18) with code 0
workbench.main.js:1449 DEBUG Terminal process exit (id: 18) state 5
workbench.main.js:1449 TRACE terminalInstance#dispose (id: 18)

[roblourens]

There is a terminal to run the "install" script which terminates quickly, then another one to set up the port tunnel to the host. That log is probably not an issue. I still have no clue.

[shavvn]

I'm experiencing the same problem with Remote - WSL.

My code version: Version: 1.36.1 (user setup) Commit: 2213894ea0415ee8c85c5eea0d0ff81ecc191529 Date: 2019-07-08T22:59:35.033Z Electron: 4.2.5 Chrome: 69.0.3497.128 Node.js: 10.11.0 V8: 6.9.427.31-electron.0 OS: Windows_NT x64 10.0.18362

WSL: Ubuntu 18.04

The console events log:

[shavvn]

Ok I think I was able to reliably reproduce this problem (at least on my setup).

Open a large data file, mine was a 1 million line data file, ~30MB.

VSCode cannot tokenize the large file, and then a console event will occur:

Extension Host extensionHost.ts:202 Debugger attached.

Wait a minute or two, VS code will "attempt to reconnect".

[thijsbro]

Thanks @shavvn !!! That solves it. Have been cursing this issue for weeks now, but you've found it! Workaround: simply delete large files in the workspace! 🥇 Yoohoo! I can work from home again without having to revert to emacs!

[thijsbro]

I mean, of course this doesn't solve it: the issue remains and I would say this is a 'bug' that needs to be fixed: can't have vscode choking when there are large files in the filesystem. But at least we now know where the apparently random disconnects come from, and can work around it while we wait for a fix.

[alexdima]

So, to reproduce, you simply open a large file, wait a bit and then the disconnection dialog pops up?

[shavvn]

@alexandrudima just to clarify, I think it has to be a large file with a format VSCode (or extension) can recognize(which will trigger the tokenizer).

For example, mine was a CSV file and I have Rainbow CSV extension.

[thijsbro]

For me, the disconnects start when I build the source tree. The build contains large object files (>100MB), and I have the ccls extension running (which might create files in the .vscode directory). Should I tell vscode to ignore the object files separately? Right now they are in my .gitignore.

[thijsbro]

@alexandrudima You don't even have to open a large file. At least I didn't. I think @shavvn is right: it might be a large vscode file. I use ccls which stores some context info in the workspace. I can fairly reliably reproduce the problem, so if you need 'good' vs. 'bad' debug traces I could provide them (if you tell me how... or show me a link, or something...)

[roblourens]

It would make sense for it to be anything that keeps the remote EH busy enough for a long time so that it can't respond to the local window.

[mykter]

Small update to my issue: after a while of being "disconnected" (some hours?), it eventually gives up and pops up a modal dialogue saying I need to reload the window to reconnect. Doing so does indeed reconnect it (although this is sometimes immediately followed by the usual not-really-disconnected behaviour starting again).

This is somewhat harder to ignore, as I lose the UI state & terminal sessions I had running.

[thijsbro]

@mykter did you try a git clean? For me, the problem disappears when I throw away large object files and excutables. The source tree itself (still >170 MB, but all small files) does not seem to cause any problems.

[mykter]

@thijsbro I haven't, but I can't as I have half a gig of untracked node_modules! I'll fire up another instance (once microsoft/vscode-remote-release#1217 is fixed and I can connect with insiders again) on a tiny workspace and see if it suffers the same issues.

[mykter]

Sounds like you know what the cause of this is now, but one extra bit of info: one thing that does actually break when it says it's disconnected is port forwarding. I have to set those up again each time it "disconnects" (which is somewhat more annoying than just dismissing the prompts!).

[roblourens]

You can try this setting to automatically set up those ports each time you reconnect: "remote.SSH.defaultForwardedPorts": []

[vscodebot[bot]]

This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines.

Happy Coding!

[mykter]

@roblourens I think the needs-more-info label is out of date - I saw in another issue that you think you know the cause of this now? But if there's any more information you need from me please say, happy to help. To be clear it's still an issue I experience every day.

[thijsbro]

I also experience the Problem on a daily basis. Happy to help debugging if someone tells me what to do (don't have much experience with debugging stuff on a windows Machine).

[petrosschilling]

I am also experiencing this after setting up a firewall on the remote server.

Every 10-30mins I get the dialog trying to reconnect. After a few attempts it successfully connects. But the problem persists.

Log after disconnection

> [16:32:26.996] SSH Resolver called for "ssh-remote+dev06", attempt 2, (Reconnection) > [16:32:26.996] SSH Resolver called for host: dev06 > [16:32:26.996] Setting up SSH remote "dev06" > [16:32:26.998] Using commit id "9df03c6d6ce97c6645c5846f6dfa2a6a7d276515" and quality "insider" for server > [16:32:27.028] Running script with connection command: ssh -T -D 61407 -o ConnectTimeout=15 dev06 bash > [16:32:27.028] Install and start server if needed > [16:32:29.527] > ssh: connect to host 203.98.82.40 port 4422: Network is unreachable > [16:32:29.527] Got some output, clearing connection timeout > [16:32:29.787] "install" terminal command done > [16:32:29.787] Install terminal quit with output: ssh: connect to host 203.98.82.40 port 4422: Network is unreachable > [16:32:29.787] Received install output: ssh: connect to host 203.98.82.40 port 4422: Network is unreachable > [16:32:29.788] Resolver error: Network is unreachable > [16:32:29.788] TELEMETRY: {"eventName":"resolver","properties":{"outcome":"failure","reason":"OfflineError","askedPw":"0","askedPassphrase":"0","asked2fa":"0","askedHostKey":"0","gotUnrecognizedPrompt":"0","remoteInConfigFile":"1"},"measures":{"resolveAttempts":2,"retries":1}} > [16:32:29.789] ------ > > > > > [16:32:30.551] > packet_write_wait: Connection to 203.98.82.40 port 4422: Broken pipe > [16:32:30.815] "install" terminal command done > [16:32:30.815] Install terminal quit with output: `  packet_write_wait: Connection to 203.98.82.40 port 4422: Broken pipe > [16:32:34.811] ------ > > > > > [16:32:34.811] SSH Resolver called for "ssh-remote+dev06", attempt 3, (Reconnection) > [16:32:34.811] SSH Resolver called for host: dev06 > [16:32:34.811] Setting up SSH remote "dev06" > [16:32:34.813] Using commit id "9df03c6d6ce97c6645c5846f6dfa2a6a7d276515" and quality "insider" for server > [16:32:34.815] Running script with connection command: ssh -T -D 61409 -o ConnectTimeout=15 dev06 bash > [16:32:34.815] Install and start server if needed > [16:32:40.388] > ssh: connect to host 203.98.82.40 port 4422: Network is unreachable > [16:32:40.388] Got some output, clearing connection timeout > [16:32:40.646] "install" terminal command done > [16:32:40.646] Install terminal quit with output: ssh: connect to host 203.98.82.40 port 4422: Network is unreachable > [16:32:40.646] Received install output: ssh: connect to host 203.98.82.40 port 4422: Network is unreachable > [16:32:40.647] Resolver error: Network is unreachable > [16:32:40.648] TELEMETRY: {"eventName":"resolver","properties":{"outcome":"failure","reason":"OfflineError","askedPw":"0","askedPassphrase":"0","asked2fa":"0","askedHostKey":"0","gotUnrecognizedPrompt":"0","remoteInConfigFile":"1"},"measures":{"resolveAttempts":3,"retries":1}} > [16:32:40.648] ------ > > > > > [16:32:50.662] ------ > > > > > [16:32:50.662] SSH Resolver called for "ssh-remote+dev06", attempt 4, (Reconnection) > [16:32:50.662] SSH Resolver called for host: dev06 > [16:32:50.662] Setting up SSH remote "dev06" > [16:32:50.662] Using commit id "9df03c6d6ce97c6645c5846f6dfa2a6a7d276515" and quality "insider" for server > [16:32:50.666] Running script with connection command: ssh -T -D 61414 -o ConnectTimeout=15 dev06 bash > [16:32:50.666] Install and start server if needed > [16:32:56.316] > ssh: connect to host 203.98.82.40 port 4422: Network is unreachable > [16:32:56.316] Got some output, clearing connection timeout > [16:32:56.577] "install" terminal command done > [16:32:56.577] Install terminal quit with output: ssh: connect to host 203.98.82.40 port 4422: Network is unreachable > [16:32:56.577] Received install output: ssh: connect to host 203.98.82.40 port 4422: Network is unreachable > [16:32:56.578] Resolver error: Network is unreachable > [16:32:56.578] TELEMETRY: {"eventName":"resolver","properties":{"outcome":"failure","reason":"OfflineError","askedPw":"0","askedPassphrase":"0","asked2fa":"0","askedHostKey":"0","gotUnrecognizedPrompt":"0","remoteInConfigFile":"1"},"measures":{"resolveAttempts":4,"retries":1}} > [16:32:56.578] ------ > > > > > [16:33:06.972] ------ > > > > > [16:33:06.972] SSH Resolver called for "ssh-remote+dev06", attempt 5, (Reconnection) > [16:33:06.972] SSH Resolver called for host: dev06 > [16:33:06.972] Setting up SSH remote "dev06" > [16:33:06.974] Using commit id "9df03c6d6ce97c6645c5846f6dfa2a6a7d276515" and quality "insider" for server > [16:33:06.975] Running script with connection command: ssh -T -D 61417 -o ConnectTimeout=15 dev06 bash > [16:33:06.975] Install and start server if needed > [16:33:13.110] > ssh: connect to host 203.98.82.40 port 4422: Network is unreachable > [16:33:13.110] Got some output, clearing connection timeout > [16:33:13.370] "install" terminal command done > [16:33:13.370] Install terminal quit with output: ssh: connect to host 203.98.82.40 port 4422: Network is unreachable > [16:33:13.370] Received install output: ssh: connect to host 203.98.82.40 port 4422: Network is unreachable > [16:33:13.370] Resolver error: Network is unreachable > [16:33:13.371] TELEMETRY: {"eventName":"resolver","properties":{"outcome":"failure","reason":"OfflineError","askedPw":"0","askedPassphrase":"0","asked2fa":"0","askedHostKey":"0","gotUnrecognizedPrompt":"0","remoteInConfigFile":"1"},"measures":{"resolveAttempts":5,"retries":1}} > [16:33:13.371] ------ > > > > > [16:33:23.391] ------ > > > > > [16:33:23.391] SSH Resolver called for "ssh-remote+dev06", attempt 6, (Reconnection) > [16:33:23.391] SSH Resolver called for host: dev06 > [16:33:23.391] Setting up SSH remote "dev06" > [16:33:23.393] Using commit id "9df03c6d6ce97c6645c5846f6dfa2a6a7d276515" and quality "insider" for server > [16:33:23.396] Running script with connection command: ssh -T -D 61422 -o ConnectTimeout=15 dev06 bash > [16:33:23.396] Install and start server if needed > [16:33:29.052] > ssh: connect to host 203.98.82.40 port 4422: Network is unreachable > [16:33:29.052] Got some output, clearing connection timeout > [16:33:29.309] "install" terminal command done > [16:33:29.309] Install terminal quit with output: ssh: connect to host 203.98.82.40 port 4422: Network is unreachable > [16:33:29.309] Received install output: ssh: connect to host 203.98.82.40 port 4422: Network is unreachable > [16:33:29.309] Resolver error: Network is unreachable > [16:33:29.310] TELEMETRY: {"eventName":"resolver","properties":{"outcome":"failure","reason":"OfflineError","askedPw":"0","askedPassphrase":"0","asked2fa":"0","askedHostKey":"0","gotUnrecognizedPrompt":"0","remoteInConfigFile":"1"},"measures":{"resolveAttempts":6,"retries":1}} > [16:33:29.310] ------ > > > > > [16:33:39.325] ------ > > > > > [16:33:39.325] SSH Resolver called for "ssh-remote+dev06", attempt 7, (Reconnection) > [16:33:39.325] SSH Resolver called for host: dev06 > [16:33:39.325] Setting up SSH remote "dev06" > [16:33:39.326] Using commit id "9df03c6d6ce97c6645c5846f6dfa2a6a7d276515" and quality "insider" for server > [16:33:39.328] Running script with connection command: ssh -T -D 61425 -o ConnectTimeout=15 dev06 bash > [16:33:39.328] Install and start server if needed > [16:33:44.908] > ssh: connect to host 203.98.82.40 port 4422: Network is unreachable > [16:33:44.908] Got some output, clearing connection timeout > [16:33:45.164] "install" terminal command done > [16:33:45.165] Install terminal quit with output: ssh: connect to host 203.98.82.40 port 4422: Network is unreachable > [16:33:45.165] Received install output: ssh: connect to host 203.98.82.40 port 4422: Network is unreachable > [16:33:45.165] Resolver error: Network is unreachable > [16:33:45.166] TELEMETRY: {"eventName":"resolver","properties":{"outcome":"failure","reason":"OfflineError","askedPw":"0","askedPassphrase":"0","asked2fa":"0","askedHostKey":"0","gotUnrecognizedPrompt":"0","remoteInConfigFile":"1"},"measures":{"resolveAttempts":7,"retries":1}} > [16:33:45.166] ------ > > > > > [16:33:55.179] ------ > > > > > [16:33:55.179] SSH Resolver called for "ssh-remote+dev06", attempt 8, (Reconnection) > [16:33:55.179] SSH Resolver called for host: dev06 > [16:33:55.179] Setting up SSH remote "dev06" > [16:33:55.180] Using commit id "9df03c6d6ce97c6645c5846f6dfa2a6a7d276515" and quality "insider" for server > [16:33:55.182] Running script with connection command: ssh -T -D 61430 -o ConnectTimeout=15 dev06 bash > [16:33:55.182] Install and start server if needed > [16:34:03.769] > ssh: connect to host 203.98.82.40 port 4422: Network is unreachable > [16:34:03.769] Got some output, clearing connection timeout > [16:34:04.308] "install" terminal command done > [16:34:04.308] Install terminal quit with output: ssh: connect to host 203.98.82.40 port 4422: Network is unreachable > [16:34:04.308] Received install output: ssh: connect to host 203.98.82.40 port 4422: Network is unreachable > [16:34:04.308] Resolver error: Network is unreachable > [16:34:04.309] TELEMETRY: {"eventName":"resolver","properties":{"outcome":"failure","reason":"OfflineError","askedPw":"0","askedPassphrase":"0","asked2fa":"0","askedHostKey":"0","gotUnrecognizedPrompt":"0","remoteInConfigFile":"1"},"measures":{"resolveAttempts":8,"retries":1}} > [16:34:04.309] ------ > > > > > [16:34:12.697] ------ > > > > > [16:34:12.697] SSH Resolver called for "ssh-remote+dev06", attempt 9, (Reconnection) > [16:34:12.697] SSH Resolver called for host: dev06 > [16:34:12.697] Setting up SSH remote "dev06" > [16:34:12.698] Using commit id "9df03c6d6ce97c6645c5846f6dfa2a6a7d276515" and quality "insider" for server > [16:34:12.699] Running script with connection command: ssh -T -D 61434 -o ConnectTimeout=15 dev06 bash > [16:34:12.699] Install and start server if needed > [16:34:14.193] > tput: No value for $TERM and no -T specified > [16:34:14.193] Got some output, clearing connection timeout > [16:34:14.195] > tput: No value for $TERM and no -T specified > [16:34:14.198] > tput: No value for $TERM and no -T specified > [16:34:14.214] > tput: No value for $TERM and no -T specified > > tput: No value for $TERM and no -T specified > > tput: No value for $TERM and no -T specified > > tput: No value for $TERM and no -T specified > [16:34:14.216] > tput: No value for $TERM and no -T specified > [16:34:14.223] > tput: No value for $TERM and no -T specified > > tput: No value for $TERM and no -T specified > [16:34:14.226] > tput: No value for $TERM and no -T specified > [16:34:14.230] > tput: No value for $TERM and no -T specified > [16:34:14.233] > tput: No value for $TERM and no -T specified > [16:34:14.241] > tput: No value for $TERM and no -T specified > > tput: No value for $TERM and no -T specified > [16:34:14.299] > 74cf0e817385: running > [16:34:14.372] > Acquiring lock on /root/.vscode-server-insiders/bin/9df03c6d6ce97c6645c5846f6dfa2a6a7d276515/vscode-remote-lock.9df03c6d6ce97c6645c5846f6dfa2a6a7d276515 > [16:34:14.403] > Found existing installation at /root/.vscode-server-insiders/bin/9df03c6d6ce97c6645c5846f6dfa2a6a7d276515... > [16:34:14.482] > Found running server... > > > > * > > * Reminder: You may only use this software with Visual Studio family products, > > * as described in the license (https://go.microsoft.com/fwlink/?linkid=2077057) > > * > > > [16:34:14.520] > Checking server status on port 45055 with wget > [16:34:14.544] > 74cf0e817385: start > > sshAuthSock==== > > agentPort==45055== > > webViewServerPort==== > > osReleaseId==centos== > > arch==x86_64== > > webUiAccessToken==== > > 74cf0e817385: end > [16:34:14.544] Received install output: > sshAuthSock==== > agentPort==45055== > webViewServerPort==== > osReleaseId==centos== > arch==x86_64== > webUiAccessToken==== > > [16:34:14.544] Remote server is listening on port 45055 > [16:34:14.544] Parsed server configuration: {"agentPort":45055,"osReleaseId":"centos","arch":"x86_64","webUiAccessToken":"","sshAuthSock":""} > [16:34:14.545] Starting forwarding server. localPort 61436 -> socksPort 61434 -> remotePort 45055 > [16:34:14.545] Forwarding server listening on 61436 > [16:34:14.545] Waiting for ssh tunnel to be ready > [16:34:14.549] [Forwarding server 61436] Got connection 0 > [16:34:14.550] Tunneled remote port 45055 to local port 61436 > [16:34:14.550] Resolved "ssh-remote+dev06" to "127.0.0.1:61436" > [16:34:14.550] TELEMETRY: {"eventName":"resolver","properties":{"outcome":"success","osReleaseId":"centos","arch":"x86_64","askedPw":"0","askedPassphrase":"0","asked2fa":"0","askedHostKey":"0","remoteInConfigFile":"1","gotUnrecognizedPrompt":"0"},"measures":{"resolveAttempts":9,"retries":1}} > [16:34:14.550] ------ > > > > > [16:34:14.560] [Forwarding server 61436] Got connection 1 > [16:34:15.175] [Forwarding server 61436] Got connection 2

[roblourens]

Looks like a different issue @petrosschilling, in your case you get Network is unreachable so it just sounds like you're offline.

[vscodebot[bot]]

This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines.

Happy Coding!

[mykter]

@roblourens same comment as last time

I think the needs-more-info label is out of date - I saw in another issue that you think you know the cause of this now? But if there's any more information you need from me please say, happy to help. To be clear it's still an issue I experience every day.

[mcswell]

I think I'm having the same issue. It's reproducible, and it happens whenever I try to open a large file (happens to be XML, 1.5M lines, 57M bytes) via sftp. After the file opens, VS reports that the SSH has disconnected. Afaict, although, this is not true; if I open a terminal, it works just fine.

BTW, the file in question is proprietary, so I cannot share it. But I suppose I could create such a file with random contents.

Let me know if you need more information (logs or s.t.).

[forrestjacobs]

I'm getting the same issue when trying to open the XML file in http://www.edrdg.org/kanjidic/kanjidic2.xml.gz (filed as microsoft/vscode-remote-release#1824, thanks to @mcswell for pointing me to this ticket.) I think anyone trying to reproduce this bug is free to download and test with that file (see http://www.edrdg.org/wiki/index.php/KANJIDIC_Project#Copyright_and_Permissions )

[mykter]

I think this might be fixed! I had a whole day with no occurrences yesterday.

[mcswell]

I can confirm that it appears to be much better, although not perfect. I am able to open the large files that I couldn't open before without it always complaining to me that it had lost the connection, although there is still some flakiness associated. I would have to experiment with it more to be more specific about that "flakiness", although one thing I notice is that it asks me more than once for the passkey. That is, it immediately asks me for the passkey when it starts up (which is expected), but then asks me again after 60 seconds or so of downloading the large file (which is unexpected). But in general, much better--at least it doesn't close down when it gets done.

[dobos]

I think this might be fixed! I had a whole day with no occurrences yesterday.

Are you on the insider version?

I'm having the same issue with a python project not particularily large. It is maybe the pycache folders that are somewhat heavy but they should be ignored anyway. On the other hand, I use the port forwards established by the same SSH connection to view tensorboard and jupyter running on the same linux server I'm accessing remotely. Also, I use a proxy node with the ProxyCommand ssh option. vscode is running on windows.

[dobos]

Interestingly, I see the error more often when I'm saving large files from a program that's running inside the integrated terminal on the remote server. The files are written into a directory outside the workspace.

[mykter]

No, I'm on stable.

My phantom disconnects weren't connected with opening large files, so it's possible we were on different underlying bugs.

To confirm my comment from a month ago though: this has completely gone away for me now. No issues this year despite no changes in my setup that I can think of.

[dobos]

Just a few more details on this issue, I hope these will help pinpointing the problem. I'm pretty sure disconnects are happening when I work with large files but it's more likely to be associated with heavy network use of the server because it also happens if I run programs that use heavy IO outside the integrated terminal. Most of my IO goes to a network-attached ceph cluster. I have a 1g connection to the cluster which is clearly a bottleneck for network file IO.

[BreezeMaxWeb-NS]

This keeps happening to me, I have a 1.6 MB log file with 170k lines, and every time I open it I get disconnected from the server within about 10 seconds after loading it.

If I don't close the file, it just keeps loop disconnecting.

[alphapuggle]

Can confirm this is still an issue. Experiencing frequent disconnects on my Pi2B+ Running Ubuntu 20.04 connecting with VS Code 1.48.0 Stable on Windows 10 Pro build 19042.450

[alexdima]

We have two connections to the remote machine:

• one is a management connection (for FS, extension installation, etc.)
• the other one is a connection to the remote extension host

We detect disconnection through a number of ways, like if the socket is closed by the OS. But the OS does not always close a broken socket. So we need to detect disconnection also by ourselves. We do this by sending a heart-beat message every 5 seconds on those connections. If a heart-beat message is not received for 20 seconds from any one of the two connections, then we enter the reconnection loop.

The remote extension host connection is susceptible to the following problem: If an extension that executes remotely freezes the remote extension host event loop for 20s, then our code that sends the heart-beat does not have a chance to execute, so the UI side will see that 20s have passed and no communication was received from the remote extension host connection and it will enter the reconnection loop.

So the root cause is IMHO always a specific extension that is implemented in a flawed way. This exact thing happens if everything is executed locally, so it is not a problem specific to remoting. The difference is that in the local case, we just silently profile the extension host when this happens and create a telemetry event with the worst offender at the time, and we don't show anything in the UI.

[airtonix]

Really speaks to the fact that the extension bisect feature needs to work properly with remote-ssh extension... currently it just sits in a loop of

1. start bisect
2. detects you're on a remote connect, and asks if the extension should be enabled? [YES]
3. reloads the window and ....
4. starts back at 1 again forgetting your answer to 2

[alexdima]

I agree, I've detailed this case in https://github.com/microsoft/vscode/issues/113568

[alexdima]

@airtonix Actually, this has already been done in https://github.com/microsoft/vscode/issues/112473 . So it is already available in Insiders and will become available in Stable that extension bisect will not disable remote extensions.

[VALLIS-NERIA]

1.60.0 win x64 Remote-SSH 0.65.7

• all other extension disabled
• right-click on a large file (110MB~)
• "delete permanently"
• begins to reconnect loop after a while

[stalkermustang]

any updates? As a full remote developer, I stuck with this so many times. That's a real problem. Maybe read just first several KB's for the show? not the whole file?

[tigerinus]

This issue has been randomizing our work on remote SSH development environment starting since about 3~4 weeks ago. The team has to switch back to their own local VSCode for now. We are a startup that embraces BYOD and all working from home, but some of the team member’s computers aren’t as powerful as the remote SSH environment, so this workaround is really just temporary.