DNS should default to local Docker host settings

[jpds]

I'm running Drone within Kubernetes on Azure, and by default it seems keen to use Google DNS for the docker build container:

time="2018-09-11T14:15:45.926616198Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
time="2018-09-11T14:15:45.926647397Z" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"

This then fails at:

fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz: temporary error (try again later)
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz: temporary error (try again later)

If I set custom_dns to my kube-dns (10.0.0.10) service, everything works fine. Docker itself defaults to the host DNS settings:

https://docs.docker.com/config/containers/container-networking/#dns-services

I think this plugin should follow the default Docker behavior instead of defaulting to Google.

[jpds]

The DinD container on the agent itself has:

$ kubectl exec -it -n testing-drone drone-agent-64dd9fdf59-967rr -c drone-dind -- sh
/ # cat /etc/resolv.conf 
nameserver 10.0.0.10
search testing-drone.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

[jpds]

The above is with 17.12.0-ce-dind from the Helm chart here: https://github.com/helm/charts/tree/master/stable/drone/

Testing with 18.06.1-ce-dind now.

[jpds]

This needs #194 to update the Docker version used by the plugin.

[bradrydzewski]

I think this plugin should follow the default Docker behavior instead of defaulting to Google.

This plugin does follow the default docker behavior.

The reason that the agent and Docker container have different dns configurations are because they are attached to different networks. Your agent is using the default network driver (bridge or whatever), however, all drone pipeline steps are attached to a user-defined network:

docker network create foo
docker run --network=foo plugins/docker

The user-defined networks in Docker using an embedded DNS server [1] and therefore have different rules for DNS configuration than the default bridge network. So there really isn't anything we can do in this plugin, but you may be able to alter the host machine configuration in some way to ensure the correct DNS server is used inside the plugin (I'm not sure, though, this is outside my area of expertise).

Alternatively you can use the global environment variables Enterprise feature [2] and set PLUGIN_CUSTOM_DNS as a global environment variable, so that you do not need to set the value in your yaml file.

[1] https://docs.docker.com/v17.09/engine/userguide/networking/configure-dns/ [2] http://docs.drone.io/configure-global-environment/

[jpds]

This plugin does follow the default docker behavior.

Sure, this is what I agree should be happening, but none of my other Kubernetes containers are doing this and suddenly defaulting to going to Google DNS.

The person at #176 says that they had (undefined) issues with yum which echos the issues I'm seeing with apk in the DinD container, so perhaps the upgrade would fix this (I'm currently deploying a image of this plugin in my private repo to test this theory).

[c2h5oh]

Btw I'm seeing the same without kubernetes. Docker container started in dind (I know it's not the best idea) also default to goodle dns and is unable to resolve pipeline services. - this is on 1.0 RC4

[sneak]

I'm running into this issue today. It seems to me a major security issue to leak DNS hostname requests unencrypted to a third party server that the user never selected or configured! My LAN DNS server encrypts all DNS traffic to the WAN; using unencrypted DNS to unauthorized servers is incorrect behavior.

root@las1:~# docker images | grep plug
plugins/docker                      latest              29c70b6af6af        2 months ago        182MB
root@las1:~# docker rmi plugins/docker
Untagged: plugins/docker:latest
Untagged: plugins/docker@sha256:1d4743c50cfe06e84962a45337dc7ac200982c00c98d67852b54bf9475d37358
Deleted: sha256:29c70b6af6af2fb454b19ce8f7b51b35a5655c8398b6dd76938f6184ff7fb05a
Deleted: sha256:2b52935f5ac6ecad710c8ec05acfd172929022010e2fce6feee1d9c967f46013
Deleted: sha256:9bceba7972a6ac6471f2515c3dbf869abd8ff74964bb5ecdae340ad4554f7653
Deleted: sha256:f9d5a45945e1f93e2a2e8a81ac440f0309409d836d119de67dbd61572f7e8c2c
Deleted: sha256:01a062ec6974efa55446c81a9276f20d27d3635fe025d071a6637cfc2dfb13dc
Deleted: sha256:6a07e4cab25962d6b819f0508607d0c8d98e91121a83dbfbd6aec202eebded77
Deleted: sha256:e1bc276529a5de677bb7c41f28c76fce259b69fe35de7a8d1e443fc630c185e7
Deleted: sha256:6789d66fedded194adfa52ade2cf79aa4f84a244b58c2e2e57a85c5883f44b2d
Deleted: sha256:649f547f80b238767bc19dea8cf242adc2fa7164d3aff47b5e2a45dd873bdd6e
Deleted: sha256:1ee98fa99e925362ef980e651c5a685ad04cef41dd80df9be59f158cf9e52951
Deleted: sha256:78c8e55f8cb4c661582af874153f88c2587a034ee32d21cb57ac1fef51c6109e
Deleted: sha256:7bff100f35cb359a368537bb07829b055fe8e0b1cb01085a3a628ae9c187c7b8
root@las1:~# date -u
Wed Feb 12 06:01:13 UTC 2020
root@las1:~#

... start build, wait a sec ...

root@las1:~# docker ps | head -2
CONTAINER ID        IMAGE                               COMMAND                  CREATED             STATUS                PORTS                                                                      NAMES
4bceb265b144        plugins/docker:latest               "/usr/local/bin/dock…"   7 seconds ago       Up 3 seconds          2375/tcp                                                                   drone-xl5Q9qFh4kAv2TI9KEp8
root@las1:~#

root@las1:~# docker exec -ti drone-xl5Q9qFh4kAv2TI9KEp8 /bin/sh -c "docker run -ti ubuntu /bin/bash -c 'cat /etc/resolv.conf'"
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
5c939e3a4d10: Pull complete
c63719cdbe7a: Pull complete
19a861ea6baf: Pull complete
651c9d2d6c4f: Pull complete
Digest: sha256:8d31dad0c58f552e890d68bbfb735588b6b820a46e459672d96e585871acc110
Status: Downloaded newer image for ubuntu:latest

search CORRECT-DHCP-DOMAIN-NAME-REDACTED
options ndots:0

nameserver 8.8.8.8
nameserver 8.8.4.4

root@las1:~# docker exec -ti drone-xl5Q9qFh4kAv2TI9KEp8 /bin/sh -c "docker info | grep -i version"
Server Version: 18.09.0
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Kernel Version: 4.15.0-76-generic
WARNING: No swap limit support
root@las1:~#

[ashwilliams1]

This plugin uses the official docker-in-docker images as its base image, which starts docker-in-docker with default settings. The docker-in-docker deaemon is responsible for default DNS settings. If the docker-in-docker daemon is not using the correct settings or is not inheriting the settings from the parent container, and you think it should do so automatically, you should open an issue with the moby / docker team since they maintain the code responsible for DNS.

Alternatively you can configure custom DNS for the docker-in-docker plugin by globally setting the PLUGIN_CUSTOM_DNS variable. You can globally set environment variables for plugins using DRONE_RUNNER_ENVIRON.

[tboerger]

IMHO this is a wontfix for this plugin, we are using the defaults of docker.

If this is an issue for you report it upstream.

[mildebrandt]

While passing the issue upstream is the easy thing to do, I don't think it's the right thing in this case. People writing build scripts and using this plugin shouldn't need to be concerned about which DNS is being used. They expect each plugin to behave the same....if a network call works in one plugin, it should work in another.

The fact that this is performing a docker-in-docker is an implementation detail that the user should be isolated from knowing. The build engineer just wants to build a container.

If the solution is to set the PLUGIN_CUSTOM_DNS environment variable, that should be front and center in big bold letters as a prerequisite to using this plugin. If not, every new user that runs into this issue has to hop on Google and figure out why their container fails to build due to a server not being found.

Ideally, the plugin would pass its DNS settings into the child container every time.

[ashwilliams1]

ideally the plugin would pass its DNS settings into the child container every time.

how? If it were straight forward then I wonder why docker is not already doing this?

[bradrydzewski]

While passing the issue upstream is the easy thing to do, I don't think it's the right thing

This assumes there is something we can do in this codebase to solve this problem. It is not yet clear to me that we can solve this issue without docker's help but I am certainly open to suggestions.

For reference this is the existing issue for dns issues in docker-in-docker: https://github.com/moby/moby/issues/20037

[mildebrandt]

The issue stems from the fact that drone uses a custom network which kicks-in docker's DNS service which isn't available to dind containers.

I also see that Drone treats certain containers as special already, including this one: https://github.com/drone/drone/blob/d84b79f027247284d26066041b1521414f68c631/operator/runner/runner.go#L255

So, how about creating another special category which passes the host's DNS settings into a plugin via --dns and related switches? That way the child containers will get the proper settings since the nameserver wouldn't be a local address.

[blopker]

I'd also be interested in a Docker plugin that uses the host's daemon instead of docker-in-docker. For self hosted deployments this would have a few benefits, like having a persistent layer cache and it would be easier to configure /etc/docker/daemon.json.

[bradrydzewski]

@blopker you can use this plugin with the host machine docker socket but there are security implications in doing so (example: malicious user overwrites an image in the host cache). Since you are running inside the firewall with trusted users, you could enable this by:

- name: publish
  image: plugins/docker
  settings:
    daemon_off: true
    purge: false
    ...
  volumes:
  - name: dockersock
     path: /var/run/docker.sock

volumes:
  - name: dockersock
    host:
      path: /var/run/docker.sock

The daemon_off setting prevents the docker in docker daemon from starting and the purge flag prevents the system from running the prune command which would otherwise clear the host machines docker cache.

The only caveat is this plugin is built for docker-in-docker and we make no guarantees that mounting the host machine socket will continue to work in the future. We only test and guarantee support for docker-in-docker. With that being said it is probably safe to use this feature because we make very few changes to this plugin, as we are more concerned with stability over features.

I also want to point out that we hope to eventually support built-in syntax for building and publishing images using the host machine daemon. We created a working prototype for building images in a safe manner, however, we need Docker to complete integration with containerd image management to allow for safe and efficient pushes. We are tracking this at https://github.com/drone/drone/issues/2462

---

@mildebrandt one concern I have is that by automatically trying to configure dns we could break existing Drone installations that do not expect this behavior. For example what happens if the host machine is using dnsmasq on a local address that cannot be reached from inside the container? Would trying to override the docker defaults cause the plugin to fail when it is otherwise working? Are there other edge cases we need to consider?

We have to be careful because there are thousands of active Drone installations and most of them are using this plugin without issue. For this reason we take a very conservative approach to changes unless we can verify the changes are not breaking.

For this reason I think any changes we make need to be opt-in for now.

The issue stems from the fact that drone uses a custom network which kicks-in docker's DNS service which isn't available to dind containers.

I did want to mention that this issue was much more common for users running Drone on kubernetes. We have since launched the native kubernetes runner which solves the problem by running pipeline steps inside pods that uses the default kubernetes network (not a custom docker network).

[mildebrandt]

Thanks for the detailed response, I understand. I didn't think about the dnsmasq issue earlier. I agree using the host machine's docker daemon could be a good solution too and will be following the linked issue.

I just tested something locally that seems to work. How feasible would it be to use host networking for the dind container, e.g. --network host from the command line? Using that, I'm able to resolve all our internal DNS entries.

Thoughts?

[bradrydzewski]

I just tested something locally that seems to work. How feasible would it be to use host networking for the dind container, e.g. --network host from the command line? Using that, I'm able to resolve all our internal DNS entries.

I really like this idea. You may also be able to use bridge instead of host.

You can override the default network settings in the yaml (below example). I cannot think of any reason this plugin would need to be attached to a user-defined network as opposed to host or bridge, so this could be a great solution to this issue.

steps:
- name: test
  image: golang
  commands:
  - go build
  - go test
  network_mode: bridge

I would suggest we start with support for DRONE_DIND_NETWORK=host|bridge to automatically override the network setting for this plugin and others like it. This is something we could implement pretty quickly.

I think we could enable by default once we have had some of our community members test this setting. Specifically I would be interested in our Gitea users who tend to install everything on the same machine (Drone, Gitea, Harbor, etc) and make very heavy use of user-defined networks. I want to make sure we don't adversely impact these types of setups.

[mildebrandt]

That's great, thanks for being open to suggestions. :)

Just to be clear, the plugin itself can still be in a user-defined network. It's the build step inside the container that would be given the host network setting. Something like:

docker build --network host -t myrepo:1.1 .

So it may affect even less than you think, and could be a set in the plugin itself rather than on Drone.

[sneak]

One caveat to the workaround specified above (network_mode: bridge in the step yaml) is that it requires that the project be marked as Trusted (network_mode changes are not allowed otherwise).

Seems to be working right now, using the correct DNS and not leaking my hostnames to Google.

Looking forward to being able to set the setting on the runner via environment instead of in the .drone.yml and re-marking the project as non-trusted.

Thanks for the workaround!

[ashwilliams1]

We could safely loosen these restrictions and only require trusted mode when someone tries to set network_mode to a value other than bridge. It never really needed to be this strict.