Connect using Docker Machine not working

[arniwesth]

Environment:

• VSCode Version: 1.40.0-insider
• OS Version: Linux Mint 19.2
• Does this issue occur when all extensions are disabled?: Not applicable

Steps to Reproduce:

$ eval $(docker-machine env remote-docker-host)
$ code-insiders
-> Docker -> Right click on running container -> View logs

Result:

$ docker logs -f f6b1eb765d24dc981ee2fe56200d3fea9089e70dee5c53ac6abb5584045f46c6
Error: No such container: f6b1eb765d24dc981ee2fe56200d3fea9089e70dee5c53ac6abb5584045f46c6

The other options (Attach, Inspect, etc ...) has the same result.

Exected result:

I am trying to do this:

https://code.visualstudio.com/docs/remote/containers-advanced#_option-2-connect-using-docker-machine

Either I am missing something or it does not work as expected.

[vscodebot[bot]]

(Experimental duplicate detection) Thanks for submitting this issue. Please also check if it is already covered by an existing one, like:

• Show logs inside remote docker's container report error (#75611)

[arniwesth]

I suspect the environment is not set correctly within VS Code. The remote docker containers are shown correctly in the "Remote Explorer" and "Docker" extension views but the VS Code terminal shows the local docker containers, i.e

docker container ls

shows the local containers.

[bwateratmsft]

@arniwesth Is there any value set for docker.host in your VSCode settings?

[bwateratmsft]

Additionally, what version of the Docker extension do you have? I was able to do the above successfully with 0.8.2.

[arniwesth]

It works for me now as well. Not sure why I got this error the first time.

[bwateratmsft]

Ah, glad it's working @arniwesth! There is one command, Docker Images: Run Azure CLI that I know may not properly run on the remote docker-machine in 0.8.2, but this is already fixed in the upcoming 0.9.0.

[arniwesth]

Additional information:

Docker Extension: 0.8.2

Problem:

Ok, I can attach to a remote docker already running using Option 2, but there are a few things that strike me as quite odd:

1) In Docker extension: Images: Right-click on image -> Run Results in the image being run locally - not on the remote docker-machine.

2) In the VS Code terminal

$ docker container ls

shows local running containers as well, whereas

$ eval $(docker-machine env remote-docker-host)
$ docker container ls

shows the remote running containers as expected.

[bwateratmsft]

@arniwesth What shell are you using? I'm on macOS, not Linux, but both of the above things work fine for me with bash.

Is anything set for docker.host in VSCode settings?

One additional thing to check. Both Run and Run Interactive use the --rm flag, so the container will be removed as soon as it stops, so if you then do docker ps in the remote machine it not list it if it is already removed. With Run, many containers stop immediately so would be removed very quickly. With Run Interactive, as soon as you Ctrl+C out of that interactive terminal, or exit VSCode, etc. it will be removed.

[arniwesth]

I use bash. docker.host is empty (as I suppose it should be?). I also updated VSCode to 1.40.0.

This workflow works for me:

Run from a terminal:

$ eval $(docker-machine env remote-docker-host)
$ code

In VSCode terminal:

$ eval $(docker-machine env remote-docker-host)

Now I can start docker containers remotely as expected and attach VSCode to them.

If I don't run eval $(docker-machine env remote-docker-host) in VSCode terminal I still see the remote containers and images in the Docker extension view, but in the VSCode terminal docker commands are issued against local docker-machine.

[bwateratmsft]

That is strange. The fact that the right stuff is showing in the explorer view means DOCKER_HOST is propagating at least as far as our instantiation of Dockerode, but the fact that it's not working in terminal commands means it isn't getting farther than that.

What happens if you do this?

From terminal:

$ eval $(docker-machine env remote-docker-host)
$ code

In VSCode terminal (without doing eval $(...)):

$ echo $DOCKER_HOST

I see the following, and running docker ps shows the right info:

[arniwesth]

From terminal:

$ eval $(docker-machine env remote-docker-host)
$ echo $DOCKER_HOST
tcp://XXX.XXX.XXX.XXX:2376
$ code
$ echo $DOCKER_HOST
tcp://XXX.XXX.XXX.XXX:2376

In VSCode terminal:

$ echo $DOCKER_HOST

$ eval $(docker-machine env remote-docker-host)
$ echo $DOCKER_HOST
tcp://XXX.XXX.XXX.XXX:2376

So before running eval $(docker-machine env remote-docker-host) in the VSCode terminal the DOCKER_HOST is empty. In the terminal that VSCode is started from, DOCKER_HOST has the correct value both before and after running code.

Same with 1.41.0-insider (latest build).

[bwateratmsft]

That explains why it's not working in VSCode terminal, but it's odd that environment variables are not propagating from the terminal used to launch VSCode into VSCode itself. Are you using any VSCode Remoting features? The behavior in those cases is to launch the terminal on the remote side; which would explain why things are not propagating from the local side.

[arniwesth]

Not sure precisely what you mean with VSCode Remoting features? I have Remote Explore and Docker extension installed. Other than that, I have no special set-up.

[bwateratmsft]

I suspect then that this must be some Linux quirk or a VSCode bug with obtaining the environment variables. @chrmarti any ideas what might be going on here? Environment variables from parent terminal are not propagating to the VSCode process itself; or at least not to terminals created from it.

[arniwesth]

I can try and boot up another Linux distro. Might be a problem specific to Linux Mint. Attaching remote docker containers without explicitly installing ssh in the image is awsome² - so I would like to iron out the last quirks.

It will be a while before I have time for this though.

[bwateratmsft]

No problem. Hopefully we can work out what is going wrong. It is possible with VSCode settings to run that eval $(...) command automatically in each terminal created by VSCode.

[chrmarti]

@arniwesth Do you have 'Inherit Env' enabled in the settings?

If that's enabled maybe your .bashrc or .bash_profile clear the variables?

[arniwesth]

@chrmarti No, Inherit Env was not enabled. It works as expected after enabling it. Thanks for pointing it out. Perhaps it should be enabled as default?

[bwateratmsft]

It was enabled by default for me on Mac; I've definitely never turned it on myself.

[arniwesth]

For some reason, it was not enabled by default on my machine. I'm closing this. Thanks for your help.

[bwateratmsft]

Strange. The code for it made it default enabled in VSCode as well.

I do recall a similar experience I had with the Docker extension; it had a setting to truncate the registry name to 10 (the old default). I had never changed it, and even though the new default was to not truncate, my machine somehow kept the old configuration.

Addendum: the reason that the Docker explorer worked without "Inherit Env" is that the VSCode process still inherited the environment, hence Dockerode did, but terminals created by VSCode do not. This is why any commands executed through terminal would fail. With "Inherit Env" enabled, the terminals inherit the relevant DOCKER_* env vars from the VSCode process itself, and so everything works.

[arniwesth]

There are still some weird things going on here:) - Today I started a docker container on a remote machine, then

$ eval $(docker-machine env remote-docker-host)
$ code

"View Logs" and "Attach Shell" etc works for the remote container in the Docker extension, but "Attach Visual Studio Code" results in the error message:

Cannot attach to the container with id
XXX...XXX,
it no longer exists.

[bwateratmsft]

The "Attach Visual Studio Code" command is provided by the Remote - Containers extension so it may be their implementation is different. I will try out this scenario.

[arniwesth]

It worked yesterday and nothing I am aware of has changed, so I'm a bit baffled by this one.

[arniwesth]

Ok, I figured the only thing I changed was enabling Inherit Env - sure enough, disabling Inherit Env now makes "Attach Visual Studio Code" work. Of course, the other stuff now does not work out-of-the-box as discussed previously.

[bwateratmsft]

@arniwesth I was able to attach without any problems even with Inherit Env enabled. I can reproduce the same error message, if I run a container with Right Click Image > Run--because of the --rm flag it is almost instantly removed, so that error is expected.

What are you using to start the container before you do "Attach Visual Studio Code"? Also, what image are you running?

[arniwesth]

I start it remotely (so, not using VS Code) on the server like this:

docker run --runtime=nvidia -d --rm --name=pytorch -p XXXX:22 -p XXXX:XXXX -v /home/xxx:/mnt/xxx --privileged pytorch/detectron2:1.0

It certainly runs:

$ docker container ls
pytorch/detectron2:1.0   "./entrypoint.sh"   17 minutes ago      Up 17 minutes       0.0.0.0:XXXX->XXXX/tcp, 0.0.0.0:XXXX->22/tcp   pytorch

I saw this the error reported very initally when I started out trying to use this method for connecting to remote containers. But right now I can only reproduce it by enabling Inherit_Env.

The docker image is based on nvidia/cuda:10.1-cudnn7-devel.

[arniwesth]

I should perhaps add that the server runs NVIDIA Docker, but to my knowledge this should not make any difference.

[bwateratmsft]

@arniwesth One more question, can you share the output of docker context inspect (both from VSCode and from the terminal used to launch it would be great)? You can redact IP addresses / etc. but please leave something like tcp://1.2.3.4 as the protocol is important.

[arniwesth]

From terminal:

$ docker context inspect
[
    {
        "Name": "default",
        "Metadata": {
            "StackOrchestrator": "swarm"
        },
        "Endpoints": {
            "docker": {
                "Host": "tcp://1.2.3.4:2376",
                "SkipTLSVerify": false
            }
        },
        "TLSMaterial": {
            "docker": [
                "ca.pem",
                "cert.pem",
                "key.pem"
            ]
        },
        "Storage": {
            "MetadataPath": "\u003cIN MEMORY\u003e",
            "TLSPath": "\u003cIN MEMORY\u003e"
        }
    }
]

From VS Code with Inherit_Env enabled:

[
    {
        "Name": "default",
        "Metadata": {
            "StackOrchestrator": "swarm"
        },
        "Endpoints": {
            "docker": {
                "Host": "tcp://1.2.3.4:2376",
                "SkipTLSVerify": false
            }
        },
        "TLSMaterial": {
            "docker": [
                "ca.pem",
                "cert.pem",
                "key.pem"
            ]
        },
        "Storage": {
            "MetadataPath": "\u003cIN MEMORY\u003e",
            "TLSPath": "\u003cIN MEMORY\u003e"
        }
    }
]

they are identical.

[bwateratmsft]

Rats, there goes my next idea. @Chuxel do you have any ideas why "Attach Visual Studio Code" would not work when combining eval $(docker-machine env) and "Inherit Env"? I is working for me with a more simplified example but it seems like "Inherit Env" shouldn't be blocking...

[arniwesth]

@bwateratmsft I will try with a simpler setup to see if I can narrow down the issue once I get some time.

[arniwesth]

I rebooted my machine and now, of course, it works:).

I had the same error initially, so something is being cached or changed in certain situations. I will keep this issue open a little longer, while I try to replicate this behaviour. I suspect it has something to do with running the same image locally and remote, but I will look more into it.

[chrmarti]

I was just trying to reproduce and it works for me too. If you see this again, you could check the output in the Dev Containers terminal and the dev console (Help > Toggle Developer Tools) and file a bug at https://github.com/microsoft/vscode-remote-release/issues. Thanks.

[arniwesth]

Check - I will close this one.