dockerd should use 127.0.0.53 when requested

[jaymzh]

• [x] This is a bug report
• [ ] This is a feature request
• [ ] I searched existing issues before opening this one

Expected behavior

When specifying {"dns":["127.0.0.53"]} in daemon.json, dockerd would use that as the DNS server.

Actual behavior

When you do this, dockerd helpfully notes that it's ignoring your request:

INFO[2020-04-15T19:47:48.994402261-07:00] detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv.conf

Unfortunately, this is not that same... and in fact it's much worse. For example, if you VPN, resolved will intelligently route the right DNS request to the right DNS servers on the right interfaces based on their domain. That way you can just query against 127.0.0.53 and things work.

However, the back-compat /run/systemd/resolve/resolv.conf has a list of all DNS servers on all interfaces from all services. This means that if have more than 1 "home" DNS server, then the DNS servers handed to you by your VPN client will be further down the list and dockerd will never query them.

If dockerd would actually query the requested address it would avoid bugs like this one and this one.

Steps to reproduce the behavior

• set dns to 127.0.0.53
• watch it ignore that

Output of docker version:

Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b7f0
 Built:             Wed Mar 11 01:25:46 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       afacb8b7f0
  Built:            Wed Mar 11 01:24:19 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker info:

Client:
 Debug Mode: false

Server:
 Containers: 18
  Running: 0
  Paused: 0
  Stopped: 18
 Images: 1
 Server Version: 19.03.8
 Storage Driver: overlay2
  Backing Filesystem: <unknown>
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.0.0-31-generic
 Operating System: Ubuntu 18.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.32GiB
 Name: ldx-vulturus.corp.vicarious.com
 ID: 5KZR:CPJV:KHZ4:MGWP:GR3G:NMUM:62QX:OJCE:XTSZ:IQ6Z:5UEY:JGA3
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.) Ubuntu 18.04

[cpuguy83]

The issue is it's not really possible to use 127.0.0.53 from the host in the container.

[jaymzh]

This is dockerds config, right? Why cannot the daemon talk to things on the host?

Also, what about --network host containers?

[pstrzelczak]

@jaymzh thanks a lot for submitting this issue. I agree that semantics of using systemd resolver is not same to just using /run/systemd/resolve/resolv.conf as /etc/resolv.conf.

@cpuguy83 can't Docker make its embedded DNS resolver route DNS queries to systemd-resolver by default? Or provide another embedded DNS resolver with the functionality mentioned which can be enabled on demand for such a case? Or option to always use embedded DNS resolver with another option to enable mentioned functionality?

[thaJeztah]

I opened a pull request that may partially address this issue; https://github.com/moby/moby/pull/41022. However, using 127..x.x resolvers will (as mentioned) only work if a custom network is used, as the embedded DNS is disabled when using the default bridge network.

not having the embedded DNS available on the default bridge network was done for backward compatibility with older releases; I did open a proposal to deprecate that altogether and always have the embedded DNS available (https://github.com/moby/moby/issues/22295), but no decision was made on that yet.

[thaJeztah]

FWIW, the situation on Docker for Mac (https://github.com/docker/for-mac/issues/2820) may be more involved, because there's more network "magic" involved to integrate the docker daemon (running in a lightweight VM) with the host's network stack; best to discuss that on the Docker for Mac issue

[stromnet]

The workaround linked by cohoe above (https://github.com/cohoe/workstation/issues/105) solved the issue for me (i.e. have systemd-resolved listen on a secondary IP, and point docker to that). In my case, I'm using systemd-resolved to ensure certain domains are looked up using VPN'ed DNS servers

[mrnicegyu11]

The workaround linked by cohoe above (cohoe/workstation#105) solved the issue for me (i.e. have systemd-resolved listen on a secondary IP, and point docker to that). In my case, I'm using systemd-resolved to ensure certain domains are looked up using VPN'ed DNS servers

In case someone runs into this, here is the solution we went for: The workaround indeed works, but it has to be made sure that the secondary IP is reachable from inside the docker containers (duh?). Since containers are in a network namespace, IPs in local ranges on the host, such as 127.0.0.54 or the mentioned 172.17.0.1, would not work for us (you can verify this with nsenter + ping). Instead, we have made systemd-resolved expose its stub DNS server to an IP address attached to a physical network interface of the machine, which is then reachable from inside the containers, akin to how public internet IP addresses are reachable. Note that depending on your setup running a local DNS server on a physical network interface of your machine might be a security issue unless further firewalling is implemented.