Make containers use WeaveDNS as resolver

[squaremo]

Harder than it sounds. There are two requirements:

1. The plugin has to be able to put nameserver x.x.x.x in the container's /etc/resolv.conf
2. The container has to be able to reach x.x.x.x.

At the minute, there's no provision for a libnetwork driver to do anything with /etc/resolv.conf -- or rather, there is a field in the driverapi, but no machinery to do anything with it. https://github.com/docker/libnetwork/pull/212 may fix this in part, if it lands and gets things right. (Currently it attributes primacy to an endpoint, regardless of where it is used; rather, it should nominate a primary endpoint for each sandbox. But then, what happens when you remove that endpoint? Who knows)

The second problem is perhaps more tricky, since it requires the container to have another interface on which to talk to weaveDNS (or, for weaveDNS to operate differently. Somehow.)

There is one clear alternative here: using another bridge device, on which weaveDNS listens, and on which each container is given an interface. This would require the recapitulation of the libnetwork bridge driver (allocating IPs and so on).

[squaremo]

There is one clear alternative here: using another bridge device

Another alternative is to wait and see if https://github.com/docker/docker/pull/13441 ends up supporting multiple endpoints supplied to docker run; but, I am not hopeful that it will. Besides, this indirectly puts the responsibility for the proper operation of the plugin in the hands of the user.

[squaremo]

There is one clear alternative here: using another bridge device, on which weaveDNS listens, and on which each container is given an interface.

The pain with this is that it needs to work with weaveDNS; i.e., weaveDNS needs to be told to use this bridge when it is run, and probably this would not play well with deployments in which weaveDNS is used for containers not using the plugin.

[squaremo]

Adam points out that there are plans afoot to merge weaveDNS and the router into one container. Presumably this would rely on the weave container being given an interface on whichever bridge is in use -- so there is an element of chicken-and-egg to this.

[tomwilkie]

Downside of having all containers on a host share a "dns" bridge is that it would potentially break isolation between containers on different networks but on the same host.

Alternative could be to have multiple interfaces for dns, one per subnet...

[squaremo]

Downside of having all containers on a host share a "dns" bridge

Yes, although this is presently the case anyway.

Alternative could be to have multiple interfaces for dns, one per subnet

Hmmmmm this seems like a lot of trouble, although better it works than not, if it's the only viable means.

[squaremo]

@rade suggests that static routes on ethwe (in each container) and a catch-all-weave-addresses route (in the weavedns container) might work.

[squaremo]

As an experiment, I did this:

./weave launch -iprange 10.20.0.0/16
./weave launch-dns 10.254.254.1/24 -debug
./bin/docker-ns weavedns ip route add 10.20.0.0/16 dev ethwe
# now start a container on the weave network, but give it weavedns's weave address
C1=$(./weave run -ti --dns=10.254.254.1 ubuntu)
# give it a route to weavedns
./bin/docker-ns $C1 ip route add 10.254.254.1/32 dev ethwe
# try resolving something
docker exec $C1 ping www.google.com

and it seems to work. This suggests a plan: At least when weaveDNS is merged with weaver, possibly before, modify it to use its weave address and static routes on ethwe instead of the docker bridge.

Trickiness: a route on the weavedns ethwe would be needed for each subnet used by a container.

[squaremo]

@awh has done some very useful investigation into giving the weave bridge an IP and using that as the nameserver IP for containers. The motivation for doing so is that the IP will be much more stable (e.g., across restarts of weavedns).

These are the required steps:

• Pick an unused private subnet, and an IP on that subnet, for the weave bridge
• Add the IP to the weave bridge
• Add a route on the host to the (weave) subnet the containers will be on, via the weave bridge; this means the host will be able to route packets (e.g., ping replies, DNS replies) back to the containers
• Add a pre-routing DNAT rule to map the bridge IP:53 -> weaveDNS IP:53 (docker does this for us if we supply a port mapping)
• Add a route on the host to the weavedns subnet; this means the host will be able to route packets coming through the DNAT rule above (docker doesn't do this for us, it doesn't know about weave subnets)