Stops resolving and error forwarding DNS

wclr commented 8 years ago

Sometimes after such lines:

Started container '99ed36e6da255b330883883679e649583bd4fd8053675c13ab669b06c9337c1f'
2016/11/11 23:36:13 Added service: 99ed36e6da255b330883883679e649583bd4fd8053675c13ab669b06c9337c1f {bbooks_manager-backend_1 bbooks_manager-backend 172.19.0.2 -1 [backend.manager.bbooks.docker]}

It stops to resolve - output just freezes, nslookup while this tells DNS request timed out. timeout was 2 seconds. After some a minute, continues to resolve.

Quite ofeten this error in the log:

Error forwarding DNS: read udp 172.17.0.2:50358->192.168.1.1:53: i/o timeout: fatal, no more nameservers to try

The same story if DNS is 8.8.8.8. Those problems happens quite often, and make impossible to use dnsdock as proxy DNS.

Any advice or suggestions on those issues?

aacebedo commented 8 years ago

Hi. Can you give me more details about your setup (OS, ips, containers)
I've never saw this behavior yet.

wclr commented 8 years ago

Window 10, docker for windows, just couple just containers running

I've already turned on windows firewall, turned off publishing of port 53. Just use DNS 172.17.0.2 (dnsdock IP)

I think there is some internal issue with docker for windows networking, maybe this relates to this https://github.com/docker/for-win/issues/224

wclr commented 8 years ago

@aacebedo In docker for windows log I see logging:

all log is full of: [21:00:29.958][VpnKit ][Debug ] com.docker.slirp.exe: UDP 10.0.75.1:54806 -> 172.219.186.49:1280 len 301 [21:00:29.959][VpnKit ][Debug ] com.docker.slirp.exe: Socket.Datagram.input udp:10.0.75.1:54806-172.219.186.49:1280: creating UDP NAT rule [21:00:29.960][VpnKit ][Warning] com.docker.slirp.exe: dropping unexpected UDP packet sent from 10.0.75.1:54806 to 172.219.186.49:1280 (valid subnets = 192.168.65.0/28; valid sources = 0.0.0.0) [21:00:29.960][VpnKit ][Warning] com.docker.slirp.exe: dropping unexpected UDP packet sent from 10.0.75.1:54807 to 172.219.186.49:1280 (valid subnets = 192.168.65.0/28; valid sources = 0.0.0.0)

And I think this eventually leads to preventing DNS forwarding for sometime.

And also there are such messages:

[20:20:02.464][DnsUpdater     ][Info   ] Network configuration change detected
[20:20:02.465][PowerShell     ][Info   ] Run script '$(Find-NetRoute -RemoteIPAddress 8.8.8.8).InterfaceIndex[0]'...
[20:20:03.630][Database       ][Info   ] Writing com.docker.driver.amd64-linux/slirp/dns=nameserver 172.17.0.2
nameserver 8.8.8.8
[20:20:03.792][VpnKit         ][Info   ] com.docker.slirp.exe: updating search domains to 
[20:20:03.793][VpnKit         ][Info   ] com.docker.slirp.exe: updating resolvers to nameserver 172.17.0.2#53
[20:20:03.793][VpnKit         ][Info   ] nameserver 8.8.8.8#53
[20:20:03.793][VpnKit         ][Info   ] com.docker.slirp.exe: using DNS forwarders on 172.17.0.2#53; 8.8.8.8#53

[20:20:03.796][DataKit        ][Info   ] +963312262us       fs9p [ERROR] dispatcher caught Unix.Unix_error(Unix.EPIPE, "bytes_read", ""): no more requests will be handled
[20:20:03.800][Database       ][Info   ] Successfully wrote 1 value(s)
[20:21:08.201][ApiProxy       ][Info   ] proxy >> GET /v1.24/containers/dnsdock/json
[20:21:08.202][ApiProxy       ][Info   ] Dial Hyper-V socket 4ecc23e1-e79a-4d0f-8707-111143ed1a66:23a432c2-537a-4291-bcb5-d62504644739
[20:21:08.202][ApiProxy       ][Info   ] Successfully dialed Hyper-V socket 4ecc23e1-e79a-4d0f-8707-111143ed1a66:23a432c2-537a-4291-bcb5-d62504644739
[20:21:08.205][ApiProxy       ][Info   ] proxy << GET /v1.24/containers/dnsdock/json
[20:21:08.207][ApiProxy       ][Info   ] proxy >> POST /v1.24/containers/dnsdock/attach?stderr=1&stdout=1&stream=1
[20:21:08.208][ApiProxy       ][Info   ] Dial Hyper-V socket 4ecc23e1-e79a-4d0f-8707-111143ed1a66:23a432c2-537a-4291-bcb5-d62504644739
[20:21:08.209][ApiProxy       ][Info   ] Successfully dialed Hyper-V socket 4ecc23e1-e79a-4d0f-8707-111143ed1a66:23a432c2-537a-4291-bcb5-d62504644739
[20:21:08.211][ApiProxy       ][Info   ] Upgrading to raw stream

Does dockdns change some internal settings of docker? What do those lines mean:

com.docker.slirp.exe: updating resolvers to nameserver 172.17.0.2#53
nameserver 8.8.8.8#53
com.docker.slirp.exe: using DNS forwarders on 172.17.0.2#53; 8.8.8.8#53

?

I see that containers make DNS requests via dnsdock. Is it possible to disable this? (as I want to use dnsdock only to access containers by domain names from host, not from other containers).

aacebedo commented 8 years ago

Dnsdock does not change anything on docker. I don't know the detail of how docker on windows actually works (I remember they use kind of vm to run docker) Can you try the following:

Install ubuntu on a virtualbox guest on your windows 10
Install docker on the guest and pull the dnsdock image
Check if dns resolution works (using dig)
Map the port 53 of the guest on the port of your like on your host.
Try to resolve name from the windows host (I don't know if windows offers a command for that)

aacebedo commented 8 years ago

Did you update your docker settings and especially the DOCKER_OPTS given to the daemon (it is where you can specify a different dns forwarder for all docker containers executed by a given daemon) ? It can explain why the DNS requests from containers are redirected to the dnsdock container

wclr commented 8 years ago

I think I will close this for now, the problems seem to related to internal docker netwoking on docker for windows. Thanks for your suggestions.

dmouse commented 7 years ago

I'm running Fedora 24 with the latest dnsdock image, I get the same error.

docker run --name=dnsdock -e DNSDOCK_NAME=dnsdock -e DNSDOCK_IMAGE=dev -p 172.17.0.1:53:53/udp -v /var/run/docker.sock:/var/run/docker.sock aacebedo/dnsdock:latest-amd64 --nameserver=208.67.222.222:83 --nameserver=8.8.8.8:53 --domain=vm

17:59:41.351 | ERROR ▶ DNS fowarding for 'read udp 172.17.0.5:55872->208.67.222.222:83: i/o timeout' failed: trying next Nameserver...
17:59:41.351 | ERROR ▶ DNS fowarding for 'read udp 172.17.0.5:44552->208.67.222.222:83: i/o timeout' failed: trying next Nameserver...
17:59:41.352 | ERROR ▶ DNS fowarding for 'read udp 172.17.0.5:48683->208.67.222.222:83: i/o timeout' failed: trying next Nameserver...
17:59:42.351 | ERROR ▶ DNS fowarding for 'read udp 172.17.0.5:57967->208.67.222.222:83: i/o timeout' failed: trying next Nameserver...
17:59:42.353 | ERROR ▶ DNS fowarding for 'read udp 172.17.0.5:59928->208.67.222.222:83: i/o timeout' failed: trying next Nameserver...
17:59:42.545 | ERROR ▶ DNS fowarding for 'read udp 172.17.0.5:54983->208.67.222.222:83: i/o timeout' failed: trying next Nameserver...
17:59:42.545 | ERROR ▶ DNS fowarding for 'read udp 172.17.0.5:34865->208.67.222.222:83: i/o timeout' failed: trying next Nameserver...
17:59:42.800 | ERROR ▶ DNS fowarding for 'read udp 172.17.0.5:44977->208.67.222.222:83: i/o timeout' failed: trying next Nameserver...
17:59:44.839 | ERROR ▶ DNS fowarding for 'read udp 172.17.0.5:50842->208.67.222.222:83: i/o timeout' failed: trying next Nameserver...
17:59:44.865 | ERROR ▶ DNS fowarding for 'read udp 172.17.0.5:35773->208.67.222.222:83: i/o timeout' failed: trying next Nameserver...
17:59:44.865 | ERROR ▶ DNS fowarding for 'read udp 172.17.0.5:59840->208.67.222.222:83: i/o timeout' failed: trying next Nameserver...
17:59:46.601 | ERROR ▶ DNS fowarding for 'read udp 172.17.0.5:43328->208.67.222.222:83: i/o timeout' failed: trying next Nameserver...
17:59:46.601 | ERROR ▶ DNS fowarding for 'read udp 172.17.0.5:59248->208.67.222.222:83: i/o timeout' failed: trying next Nameserver...
17:59:46.601 | ERROR ▶ DNS fowarding for 'read udp 172.17.0.5:39965->208.67.222.222:83: i/o timeout' failed: trying next Nameserver...
17:59:46.601 | ERROR ▶ DNS fowarding for 'read udp 172.17.0.5:50132->208.67.222.222:83: i/o timeout' failed: trying next Nameserver...
....
18:02:27.010 | CRITICAL ▶ DNS fowarding for 'read udp 172.17.0.5:49120->8.8.8.8:53: i/o timeout' failed: no more nameservers to try

dmouse commented 7 years ago

I use this iptable rules to allow forward,

sudo iptables -A FORWARD -i docker0 -o wlp3s0 -j ACCEPT 
sudo iptables -A FORWARD -i wlp3s0 -o docker0 -j ACCEPT

aacebedo commented 7 years ago

Hi Why are you redirecting the containers port to the docker interface ? An easier way is to redirect the port 53 to the public IP of your host.this way you do have to modify your iptables

aacebedo / dnsdock

Stops resolving and error forwarding DNS #82