search

vmware / vic

vSphere Integrated Containers Engine is a container runtime for vSphere.
http://vmware.github.io/vic
Other
639 stars 173 forks source link

Nameserver entries in containerVM /etc/resolv.conf limits attached network count to 3 #5748

Open corrieb opened 7 years ago

corrieb commented 7 years ago

User Statement:

As a container developer, I may need to connect my container to multiple networks for isolating traffic between containers.

Details:

This follows on from #5738 investigations. In the case where you have a Container network and a bridge network added to a containerVM, you'll typically end up with 3 nameserver entries in resolv.conf (assuming the external network is configured with 2). That's the maximum it will allow. If you want more, you need to use a solution like dnsmasq.

This is just as true in the case where you want to attach to more than 3 internal networks

As a result, if I tried to do the following:

docker network create --internal one
docker network create --internal two
docker create --name test --net ContainerNetwork -it ubuntu
docker network connect one test
docker network connect two test
docker start -ai test

the result is a resolv.conf that looks like this:

nameserver 10.118.65.1
nameserver 10.118.65.2
nameserver 172.18.0.1
nameserver 172.19.0.1
options timeout:15
options attempts:5

The outcome is that the containerVM won't be able to resolve anything on the network "two" due to the limitation highlighted here: https://linux.die.net/man/5/resolv.conf

What's worse is that the specification of --dns-server on the command-line prepends a line to resolv.conf for each entry, so there is the high probability of --dns-server pushing name server entries for attached networks out of scope, past the limit of 3.

Acceptance Criteria:

At the very least, this limitation needs to be documented, pending a fix. What would be preferable is that we investigate and implement a solution that doesn't have these limitations.

The main user experience issue is that there's no warning as to the limitation when the container is created or started and there may well be Compose files that use a number of networks that exceed this limit which will break in ways that are difficult to for a customer to debug

chengwang86 commented 7 years ago

According to https://docs.docker.com/engine/userguide/networking/#embedded-dns-server and https://docs.docker.com/engine/userguide/networking/configure-dns/, it seems that vanilla docker has an embedded dns server 127.0.0.11 which delegates all the dns servers for different networks. 

The IP addresses passed via the --dns option is used by the embedded DNS server to forward the DNS query if embedded DNS server is unable to resolve a name resolution request from the containers. These --dns IP addresses are managed by the embedded DNS server and will not be updated in the container's /etc/resolv.conf file.
In the absence of the --dns=IP_ADDRESS..., --dns-search=DOMAIN..., or --dns-opt=OPTION... options, Docker uses the /etc/resolv.conf of the host machine (where the docker daemon runs).

My takeaway is:

chengwang86 commented 7 years ago

According to here https://github.com/vmware/vic/blob/master/vendor/github.com/docker/libnetwork/sandbox_dns_unix.go#L298, it seems like the external dns servers are stored in resolv.conf in the sandbox.

stuclem commented 7 years ago

Created https://github.com/vmware/vic-product/issues/512 to track the doc update. Removing user-doc from this one.