Upgrading to 1.63 breaks attaching to Remote Containers over SSH if you have a custom docker executable path configured

[begillil]

• VSCode Version: 1.63
• Local OS Version: Windows 10 19042.1348
• Remote OS Version: Ubuntu 18.04
• Remote Extension/Connection Type: Docker over SSH (e.g. ssh to a remote machine and attach to a container running on that machine)
• Logs:

  12265 ms] Remote-Containers 0.209.6 in VS Code 1.63.1 (fe719cd3e5825bf14e14182fddeb88ee8daf044f).
  [12264 ms] Start: Run: ssh begillil@############ /bin/sh -c /bin/sh
  [12279 ms] Start: Run in host: id -un
  [12577 ms] begillil
  [12578 ms] 
  [12579 ms] Start: Run in host: cat /etc/passwd
  [12584 ms] Start: Run in host: echo ~
  [12587 ms] /home/begillil
  [12587 ms] 
  [12588 ms] Start: Run in host: test -x '/home/begillil/.vscode-remote-containers/bin/fe719cd3e5825bf14e14182fddeb88ee8daf044f/node'
  [12590 ms] 
  [12591 ms] 
  [12591 ms] Exit code 1
  [12592 ms] Start: Run in host: test -x '/home/begillil/.vscode-server/bin/fe719cd3e5825bf14e14182fddeb88ee8daf044f/node'
  [12596 ms] 
  [12596 ms] 
  [12597 ms] Start: Run in host: test -f '/home/begillil/.vscode-server/bin/fe719cd3e5825bf14e14182fddeb88ee8daf044f/node_modules/node-pty/package.json'
  [12600 ms] 
  [12601 ms] 
  [12601 ms] Start: Run in host: test -f '/home/begillil/.vscode-remote-containers/dist/vscode-remote-containers-server-0.209.6.js'
  [12604 ms] 
  [12605 ms] 
  [12607 ms] userEnvProbe: loginInteractiveShell (default)
  [12608 ms] userEnvProbe shell: /bin/bash
  [12747 ms] userEnvProbe PATHs:
  Probe:     '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/begillil/.dotnet/tools'
  Container: None
  [12750 ms] Start: Run in Host: C:\Path\To\docker.exe version --format {{.Server.APIVersion}}
  [12756 ms] Host server: (node:29747) UnhandledPromiseRejectionWarning: Error: spawn C:\Path\To\docker.exe ENOENT
  at Process.ChildProcess._handle.onexit (internal/child_process.js:269:19)
  at onErrorNT (internal/child_process.js:465:16)
  at processTicksAndRejections (internal/process/task_queues.js:80:21)
  (Use `node --trace-warnings ...` to show where the warning was created)
  (node:29747) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
  [12757 ms] Host server: (node:29747) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
  [12759 ms] Host server: (node:29747) PromiseRejectionHandledWarning: Promise rejection was handled asynchronously (rejection id: 1)
  [12760 ms] spawn C:\Path\To\docker.exe ENOENT
  [12762 ms] CLI host's PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/begillil/.dotnet/tools
  [305448 ms] Start: Run in Host: C:\Path\To\docker.exe version --format {{.Server.APIVersion}}
  [305460 ms] Host server: (node:29747) UnhandledPromiseRejectionWarning: Error: spawn C:\Path\To\docker.exe ENOENT
  at Process.ChildProcess._handle.onexit (internal/child_process.js:269:19)
  at onErrorNT (internal/child_process.js:465:16)
  at processTicksAndRejections (internal/process/task_queues.js:80:21)
  (node:29747) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)
  [305467 ms] spawn C:\Path\To\docker.exe ENOENT
  [305468 ms] CLI host's PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/begillil/.dotnet/tools
  [305469 ms] Host server: (node:29747) PromiseRejectionHandledWarning: Promise rejection was handled asynchronously (rejection id: 2)
  [337359 ms] Start: Run in Host: C:\Path\To\docker.exe version --format {{.Server.APIVersion}}
  [337365 ms] Host server: (node:29747) UnhandledPromiseRejectionWarning: Error: spawn C:\Path\To\docker.exe ENOENT
  at Process.ChildProcess._handle.onexit (internal/child_process.js:269:19)
  at onErrorNT (internal/child_process.js:465:16)
  at processTicksAndRejections (internal/process/task_queues.js:80:21)
  (node:29747) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 3)
  [337368 ms] spawn C:\Path\To\docker.exe ENOENT
  [337369 ms] CLI host's PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/begillil/.dotnet/tools
  [337370 ms] Host server: (node:29747) PromiseRejectionHandledWarning: Promise rejection was handled asynchronously (rejection id: 3)

Steps to Reproduce:

1. Configure remote.containers.dockerPath to a custom path, e.g. C:\Path\To\docker.exe
2. Remote-SSH: Connect Current Window to Host and open a folder
3. Attempt to run Remote-Containers: Attach to Running Container...

Expected: It shows a list of containers on the remote machine to attach to Actual: A prompt saying "Docker version 17.12.0 or later is required" (just to confirm, both the local docker.exe and the remote docker installation are both newer than version 17.12.0)

The same steps did work on version 1.62.

I saw that recently #2994 was completed and made available in version 1.63, which is awesome. I've wished it worked that way since I first tried to set up this kind of remote container over ssh configuration.

However, it seems when that change was introduced, nothing accounted for the fact that existing custom docker executable paths were not likely to be available on the remote machine. So it tries to execute docker from a path that doesn't exist on the remote machine and assumes docker must not be installed when that fails.

Also, I'll add that if you do follow the "Docker version required" prompt and install docker, it will install Docker Desktop locally, but the custom path is still configured, and still not available on the remote machine, so it will still fail the same way.

I was able to fix the problem for myself by setting remote.containers.dockerPath back to the default value of docker, but it took a bit of research to figure out that that was the problem in order to get to that point.

[Chuxel]

@chrmarti Is this setting respected if it's set as a machine setting on the Remote - SSH host? Certainly that would make sense.

[486]

I am experiencing this even when the path is the default, i.e. docker. I downgraded to some arbitrary version v0.205.2 and now the issue is gone.

[chrmarti]

@486 Could you open a new issue with the log you get when this fails with the default setting? It sounds like that might be a different issue.

[chrmarti]

There is currently no way to make setting a 'machine setting' while connected with Remote-SSH and then a 'SSH server setting' while connected with Remote-Containers via that SSH server. VS Code settings are not aware of the SSH server in any way when we connect through the SSH server to the container.

We could make it a machine setting and read the SSH server's settings.json in Remote-Containers when connecting to the dev container, but then the setting would show up in the container's (and all other remotes) machine settings UI.

/cc @sandy081 for ideas.

[sandy081]

@chrmarti Not sure if I understand the scenario. Can you please elaborate this more?

[chrmarti]

@sandy081 There is a setting to configure the Docker executable's path. That setting depends on the machine, so machine scope seems appropriate. However, when we connect to a container on an SSH server, that setting would be a machine setting on the SSH server, not the container. At that point the container is the remote and VS Code does not know about the SSH server.

[sandy081]

I see, so the SSH server is not the VS Code Remote in this case? Instead the container in the SSH server is the remote? We have two remotes and we are not reading any settings from the first remote as it is not VS Code Remote. Since this is feature of remote containers, can remote containers read machine settings from SSH server?

[chrmarti]

Making it a machine setting would make it show up in the container's (and all other remotes) machine settings UI when we really only want it in the machine settings for Remote-SSH (and Remote-WSL). Maybe that's acceptable. 🤔

[sandy081]

Yeah that seems good to me

[ikivanc]

I faced with similar issue:

VSCode Version: 1.63.2 Local OS Version: Windows 11 22000.434 Docker Version: Docker Desktop 4.4.4 (73704) Remote Extension/Connection Type: Open Folder in Container

[yangm97]

Support for podman containers over ssh is broken too.