DevContainer started from docker compose breaks `docker-in-docker` networking

[paule96]

Hi devcontainers features team,

I found a weird DNS behavior in the docker-in-docker feature. So, let's begin with a meme:

So now let's start with the issue. docker-in-docker works fine in devcontainers that just use the image option to boot. So, if you .devcontainers.json looks like that:

{
    "name": "Debian",
        "image": "mcr.microsoft.com/devcontainers/base:bullseye",
    "features": {
        "ghcr.io/devcontainers/features/docker-in-docker:2": {}
    }
}

Then DNS should work fine and if you run the following command inside the devcontainer:

docker run ubuntu /bin/bash -c "cat /etc/resolv.conf"

you will get:

# DNS requests are forwarded to the host. DHCP DNS options are ignored.
nameserver 192.168.65.7

This name server is an IP that is generated by your docker desktop installation and will redirect DNS queries to the DNS server that is configured in your windows host.

If you now change the model of your devcontainer.json to use a compose file to build the devcontainer, like shown in the following example:

{
    "name": "Debian",
        "dockerComposeFile": "devcontainer.docker-compose.yml",
    "service": "devcontainer",
    "features": {
        "ghcr.io/devcontainers/features/docker-in-docker:2": {}
    }
}

you will see are diffrent behaivior then before with the same command. So, let's run the command:

docker run ubuntu /bin/bash -c "cat /etc/resolv.conf"

again, and let’s see what the output is:

options ndots:0

nameserver 8.8.8.8
nameserver 8.8.4.4

That looks quite different and can be also really bad. For example, in my company it's not allowed to make DNS request to the internet. We are forced to use the internal DNS server. So, in this situation the ubuntu container is basically offline because he can't resolve names anymore.

The expected behavior would be of course that we would have the same output as before we changed to the compose file.

hope this is enough information to get this fixed.

Best regards

Some Metainforamtion:

DemoRepo for this bug: https://github.com/paule96/DevContainerSampleBug WSL version: 1.2.5.0 Kernel version: 5.15.90.1 WSLg version: 1.0.51 MSRDC version: 1.2.3770 Direct3D version: 1.608.2-61064218 DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp Windows version: 10.0.19045.2965 Visual Studio Code: 1.78.2 (user setup) DevContainers Extension: v0.292.0

[jkeech]

Thanks for the report and for sharing a minimal repro. My first question was whether the same base image is used in both scenarios, but based on your repo, it looks like both are using mcr.microsoft.com/devcontainers/base:bullseye which checks out.

I suspect this is related to the docker networking mode used for the outer devcontainer in single container mode vs when using compose. The inner docker-in-docker inherits the networking of the parent container as far as I understand. If you add network_mode: "host" on the devcontainer service in the compose yaml file, does that fix things?

[paule96]

@jkeech yes this works. But then other containers in the compose file that should be accessible by the devcontainer can't be resolved. for example the following yaml snippet:

  influx:
    image: influxdb:2.7.1
    container_name: influx
    ports:
      - "8086:8086"

adds an influx db. This can't be accessed now by the devcontainer. The only way to change this I find for now is to also add the network_mode to influx too. But then it's not accessible with its name. It's then accessible only via the localhost of the devcontainer.

I tried to create networks but, it's for some reason of IP conflicts not possible. So, I guess this is workaround but not a good solution.

[jkeech]

If you can't use host networking, then could you configure the DNS server that you want to use? https://docs.docker.com/network/#dns-services

The DNS docs in Docker also say that it uses 8.8.8.8 by default which is probably where you see the 8.8.8.8 coming from.

--dns The IP address of a DNS server. To specify multiple DNS servers, use multiple --dns flags. If the container can’t reach any of the IP addresses you specify, it uses Google’s public DNS server at 8.8.8.8. This allows containers to resolve internet domains.

The issue here seems to be that you want to use the DNS from the host, but you don't want to use the networking stack from the host for other reasons. So manually configuring the DNS to be the same as the host is likely your best option. I don't think this is really devcontainers doing this, but just related to how you are configuring networking within the containers for your scenario. If the DNS IP on the host is not reachable from within the container, you'd likely need to find a way to expose that from the host into the container.

[paule96]

Yes, I agree that this is docker knowledge that is needed here. But it should be a warning or something in the docs, that if you use a compose file for your docker container, you maybe destroy the network stack of your child containers. I mean what I was seeing so far, it's only the DNS config that gets destroyed. So I guess your fix will work (I can only test it on Monday because I'm already at the weekend :))

But maybe we could at this to the docs somewhere. It's not really obvious when you use this feature.

[bamurtaugh]

Thanks @paule96 for sharing this. We'd welcome a docs PR like you mention!