Get ECONNRESET during launch

[sorenisanerd]

Hi. I'm trying to use the dev container feature. I'm on a Mac and I'm using "Remote - SSH" to connect to a Linux development server. It runs Oracle Linux 8. I want the containers to run on that Linux box.

My devcontainer.json is unremarkable (chose C++ + Debian 11):

// For format details, see https://aka.ms/devcontainer.json. For config options, see the README at:
// https://github.com/microsoft/vscode-dev-containers/tree/v0.217.4/containers/cpp
{
    "name": "C++",
    "build": {
        "dockerfile": "Dockerfile",
        // Update 'VARIANT' to pick an Debian / Ubuntu OS version: debian-11, debian-10, debian-9, ubuntu-21.04, ubuntu-20.04, ubuntu-18.04
        // Use Debian 11, Debian 9, Ubuntu 18.04 or Ubuntu 21.04 on local arm64/Apple Silicon
        "args": { "VARIANT": "debian-11" }
    },
    "runArgs": ["--cap-add=SYS_PTRACE", "--security-opt", "seccomp=unconfined"],

    // Set *default* container specific settings.json values on container create.
    "settings": {},

    // Add the IDs of extensions you want installed when the container is created.
    "extensions": [
        "ms-vscode.cpptools",
        "ms-vscode.cmake-tools"
    ],

    // Use 'forwardPorts' to make a list of ports inside the container available locally.
    // "forwardPorts": [],

    // Use 'postCreateCommand' to run commands after the container is created.
    // "postCreateCommand": "gcc -v",

    // Comment out to connect as root instead. More info: https://aka.ms/vscode-remote/containers/non-root.
    "remoteUser": "vscode"
}

Here is the log: https://gist.github.com/sorenisanerd/779d37ba6350acdf4d4bf6fc88d5ef0c

I'm honestly not completely sure, but I believe there's a transparent proxy of some sort sitting in between my dev box and update.code.visualstudio.com, but I've tried both wget and curl, and they're both perfectly happy connecting to it. Unless I can reproduce or at least understand the problem, I can't really take it to my network team to get it fixed.

I'm perfectly happy with a temporary workaround like, say, "download this file, put it in this directory, and it'll use that instead of trying to download it again", but I'm trying to make this work for a big team of engineers, so I need this addressed.

[bamurtaugh]

Does connecting to your remote box via SSH work when there's no dev container (through Remote-SSH and/or an external terminal not in VS Code)? Just want to try to see if the proxy issue could be from Remote-SSH, SSH in general, or Remote-Containers.

[sorenisanerd]

Remote via SSH works perfectly. Absolutely game changing feature, that one.

I just got the container thing to work. Only briefly, though.

I tried a bunch of different network things to take outbound proxies and firewalls and whatever out of the equation. No change.

Then I installed vscode on the server and forwarded it over vnc. Still no dice. I finally noticed that there is a vscode volume that was getting mounted. I deleted it and I could complete the container build and use it for dev work. That volume thing should probably go in some troubleshooting FAQ or something.

I tried connecting from my workstation and it ALSO worked now. Yay.

Before I declared success, I deleted the vscode volume again. Also deleted all the containers and images on the server, nuked my vscode, .vscode-remote-containers, .vscode-server from my home directory on the server and tried again from my desktop. It did not work.

This is the log from when I just tried again: https://gist.github.com/sorenisanerd/9ae3cd00afedfd14c30e57a762bcda05

[sorenisanerd]

I ran vscode on the server again to bootstrap it. It worked. I could also connect correctly from my desktop. To make sure it wasn't just working because I somehow shared a connection into the container between my local vscode and the vscode running on the server, I rebooted it. I can still connect from my desktop. So it's only the bootstrapping process that fails for some reason. halp :)

[sorenisanerd]

I installed the "Remote - SSH" extension in vscode on the server and did a loopback connection and then tried "reopen in container". That way, both are doing UI -> SSH -> docker. It worked.

My desktop machine IS behind a transparent proxy. It proxies https, too, so any https client needs the proxy's CA cert, otherwise it'll fail to validate the tls cert. Could that be the culprit somehow? Does it attempt to fetch anything on the client side and that's why we get ECONNRESET on the tls connection?

[bamurtaugh]

cc @chrmarti do you think this could be https://github.com/microsoft/vscode-remote-release/issues/986?

[sorenisanerd]

I think that's a different issue. The traceback I get doesn't say anything about handshake.

A lot of the time when I try to add a new extension, I get an error and I'm directed to download it manually and install from the .vsix. I'm logged in on the marketplace, so the download works for me. If I try with wget or something (from anything that doesn't have a cookie for the marketplace), I get a 429 (Too Many Requests). Presumably because we have too many vscode users coming from the same IP (the aforementioned mitm proxy).

Could this be the same? That wouldn't happen until after the handshake.

The remote.SSH.localServerDownload setting lets me choose whether to download on the server or download on the client and scp to the host. Does "Remote - Containers" always download on the client? That would explain pretty much everything.

[sorenisanerd]

I just tried installing a new extension, and I get this:

[2022-04-19 14:34:48.212] [renderer3] [error] ["Invalid: Corrupt ZIP: end of central directory record signature not found","    at W.downloadInstallableExtension (vscode-file://vscode-app/Users/e069106/Desktop/Visual%20Studio%20Code%20-%20Insiders.app/Contents/Resources/app/out/vs/code/electron-browser/sharedProcess/sharedProcessMain.js:76:218816)","    at async W.doRun (vscode-file://vscode-app/Users/e069106/Desktop/Visual%20Studio%20Code%20-%20Insiders.app/Contents/Resources/app/out/vs/code/electron-browser/sharedProcess/sharedProcessMain.js:76:217195)"]
[2022-04-19 14:34:48.233] [renderer3] [error] Corrupt ZIP: end of central directory record signature not found: Invalid: Corrupt ZIP: end of central directory record signature not found
    at W.downloadInstallableExtension (vscode-file://vscode-app/Users/e069106/Desktop/Visual%20Studio%20Code%20-%20Insiders.app/Contents/Resources/app/out/vs/code/electron-browser/sharedProcess/sharedProcessMain.js:76:218816)
    at async W.doRun (vscode-file://vscode-app/Users/e069106/Desktop/Visual%20Studio%20Code%20-%20Insiders.app/Contents/Resources/app/out/vs/code/electron-browser/sharedProcess/sharedProcessMain.js:76:217195)

which looks like part of what I got in the log when trying to build and connect to the container here:

[2022-04-19T00:48:52.729Z] Start: Run in container: (dd iflag=fullblock bs=8192 count=0 2>/dev/null; dd iflag=fullblock bs=0 count=1 2>/dev/null) | tar --no-same-owner -xz -C /vscode/vscode-server/bin/linux-x64/dfd34e8260c270da74b5c2d86d61aee4b6d56977_1650329332643
[2022-04-19T00:48:52.787Z] 
[2022-04-19T00:48:52.787Z] 
gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now
[2022-04-19T00:48:52.787Z] Exit code 2
[2022-04-19T00:48:52.787Z] Stop (58 ms): Run in container: (dd iflag=fullblock bs=8192 count=0 2>/dev/null; dd iflag=fullblock bs=0 count=1 2>/dev/null) | tar --no-same-owner -xz -C /vscode/vscode-server/bin/linux-x64/dfd34e8260c270da74b5c2d86d61aee4b6d56977_1650329332643

which makes perfect sense if the 429 isn't handled, and we just try to unpack the truncated (empty?) response.

A little further down:

[2022-04-19T00:48:52.788Z] Start: Downloading VS Code Server
[2022-04-19T00:48:52.788Z] dfd34e8260c270da74b5c2d86d61aee4b6d56977 linux-x64 stable 
[2022-04-19T00:48:52.838Z] Stop (1069 ms): Run in Host: docker exec -i -u vscode -e VSCODE_REMOTE_CONTAINERS_SESSION=6db4737c-4c0b-4532-bb63-2052f05df84a1650329308403 416837769ffd52f0d8e5319533ace5227ff5e6d578c9b07b8f29c0f4c1a8ac02 /bin/sh
[2022-04-19T00:48:52.838Z] Stop (251 ms): Run in Host: docker exec -i -u root 416837769ffd52f0d8e5319533ace5227ff5e6d578c9b07b8f29c0f4c1a8ac02 /bin/sh
[2022-04-19T00:48:52.852Z] Host server: packet-stream asked to read after closed { value: { code: 0, signal: null }, end: false, req: -81 }
packet-stream asked to read after closed { value: { code: 0, signal: null }, end: false, req: -66 }
[2022-04-19T00:48:52.899Z] Stop (23102 ms): Run: ssh devcloud /bin/sh
[2022-04-19T00:48:52.899Z] Host server terminated (code: 0, signal: null).
[2022-04-19T00:48:55.056Z] read ECONNRESET
[2022-04-19T00:48:55.056Z] Retrying to download VS Code Server.
[2022-04-19T00:48:55.275Z] Stop (2487 ms): Downloading VS Code Server
[2022-04-19T00:48:55.276Z] Start: Installing VS Code Server
[2022-04-19T00:48:55.276Z] Start: Run in container: mkdir -p /vscode/vscode-server/bin/linux-x64/dfd34e8260c270da74b5c2d86d61aee4b6d56977_1650329335276
[2022-04-19T00:48:55.276Z] Stop (0 ms): Installing VS Code Server
[2022-04-19T00:48:55.281Z] Shell server failed: Error: unexpected end of parent stream
    at sn.destroy (/Users/e069106/.vscode/extensions/ms-vscode-remote.remote-containers-0.232.6/dist/extension/extension.js:11:19161)
    at sn.write (/Users/e069106/.vscode/extensions/ms-vscode-remote.remote-containers-0.232.6/dist/extension/extension.js:11:20040)
    at /Users/e069106/.vscode/extensions/ms-vscode-remote.remote-containers-0.232.6/dist/extension/extension.js:11:23061
    at /Users/e069106/.vscode/extensions/ms-vscode-remote.remote-containers-0.232.6/dist/extension/extension.js:11:8797
    at /Users/e069106/.vscode/extensions/ms-vscode-remote.remote-containers-0.232.6/dist/extension/extension.js:11:27349
    at /Users/e069106/.vscode/extensions/ms-vscode-remote.remote-containers-0.232.6/dist/extension/extension.js:11:17587
    at n (/Users/e069106/.vscode/extensions/ms-vscode-remote.remote-containers-0.232.6/dist/extension/extension.js:11:27297)
    at /Users/e069106/.vscode/extensions/ms-vscode-remote.remote-containers-0.232.6/dist/extension/extension.js:11:27342
    at Object.cb (/Users/e069106/.vscode/extensions/ms-vscode-remote.remote-containers-0.232.6/dist/extension/extension.js:11:17698)
    at l (/Users/e069106/.vscode/extensions/ms-vscode-remote.remote-containers-0.232.6/dist/extension/extension.js:11:15701)
    at u (/Users/e069106/.vscode/extensions/ms-vscode-remote.remote-containers-0.232.6/dist/extension/extension.js:11:15963)
    at /Users/e069106/.vscode/extensions/ms-vscode-remote.remote-containers-0.232.6/dist/extension/extension.js:11:16046
    at s (/Users/e069106/.vscode/extensions/ms-vscode-remote.remote-containers-0.232.6/dist/extension/extension.js:16:5302)
    at Socket.<anonymous> (/Users/e069106/.vscode/extensions/ms-vscode-remote.remote-containers-0.232.6/dist/extension/extension.js:16:5403)
    at Socket.emit (node:events:390:28)
    at addChunk (node:internal/streams/readable:315:12)
    at readableAddChunk (node:internal/streams/readable:289:9)
    at Socket.Readable.push (node:internal/streams/readable:228:10)
    at Pipe.onStreamRead (node:internal/stream_base_commons:199:23)

...which also makes sense if we're getting a truncated response.

[bamurtaugh]

The remote.SSH.localServerDownload setting lets me choose whether to download on the server or download on the client and scp to the host. Does "Remote - Containers" always download on the client?

By default, Remote - SSH will attempt to download on the remote host, but if you enable remote.SSH.allowLocalServerDownload, the extension will fall back to downloading VS Code Server locally and transferring it remotely once a connection is established.

I believe Remote-Containers will always download the server on the remote host.

[sorenisanerd]

Hm. Then I'm at a loss :)

I've managed to work out that the URL it's trying to fetch is https://update.code.visualstudio.com/commit:dfd34e8260c270da74b5c2d86d61aee4b6d56977/server-linux-x64/stable

I can do that from the remote host, but not from my desktop.

[chrmarti]

Remote-Containers always downloads the server tar from the local machine (where the VS Code window runs). We could add downloading remotely to make it work in this setup.

[sorenisanerd]

@chrmarti That would be AMAZING. Thank you!

[Martin91]

I encountered the exactly same issue. I have to kill the vscode server manually to make it work, although I don't know why the kill helps.