DNS fails on image pull: dial tcp: lookup index.docker.io

AloisReitbauer commented 9 years ago

See error message below:

Get https://index.docker.io/v1/repositories/library/ubuntu/images: dial tcp: lookup index.docker.io on 192.168.90.1:53: read udp 192.168.90.1:53: i/o timeout

mchiang0610 commented 9 years ago

This seems to be a problem with your DNS loading an older entry.

Is this on Mac or Windows?

Try:

Click on the Docker CLI button
Type in docker-machine ssh
echo "nameserver 8.8.8.8" > /etc/resolv.conf

firrae commented 9 years ago

Hi there,

I'm having the same issue and when I follow your answer I get the following: Error: Please specify a machine name. After adding, what I assume is, the machine name I get the following: Error: Host does not exist: hello-world. I was trying to add the basic offical hello-world container to test it out.

Any further help is appreciated.

rgbkrk commented 9 years ago

If you run docker-machine ls it will show which you have available. That being said, mine is named dev (run docker-machine ssh dev).

firrae commented 9 years ago

Ah, thanks a bunch for that. Runs like a charm now.

ambled commented 8 years ago

I'm not sure if this is the/a cause or needs to be a separate issue:

my boot2docker.iso shows as: Linux dev 4.0.7-boot2docker #1 SMP Wed Jul 15 00:01:41 UTC 2015 x86_64 GNU/Linux

I've not only had this DNS issue, but it keeps reoccurring every 12 hours. I discovered that the dhcp configurations in this iso has some issues:

* when the dhcp reservation (which might be low) expires, the /etc/resolv.conf gets rewritten by udhcpc
* udhcpc uses  /usr/share/udhcpc/default.script
* default.script either clobbers and rewrites only the dhcp response, or ...
* you can override the DHCP list by creating a 'dns' environment variable entry in default.script as a whitespace seperated list of ips. However, default.script handles this case poorly and appends the configured nameservers every time the lease expires

the end result is that as configured (at least with the responses I get from my DHCP server) the VM might be working initially for people, then stops working because of a pause/resume machine event, network reconfiguration or just a DHCP lease timeout.

Since all of these events can trigger the problem and DHCP leases can vary widely between environments, it might be complicating efforts to find the root cause.

As this issue seems to come from upstream on the boot2docker.iso, my current workaround seems to be holding (bypassing the NAT DNS and forwarding to the host).

docker-machine stop dev
VBoxManage modifyvm dev --natdnsproxy1 off --natdnshostresolver1 off
docker-machine start dev

There are lots of ways to address this, not sure which makes the most sense. I think I'd like the equivalent of the resovlconf framework with head and tail entries. Improving the udhcpc script to look for /etc/resolvconf/resolvconf.d or pull settings from /etc/default seems like a possible path, however that still ends up requiring manually fiddling files on the virtual host machine.

ambled commented 8 years ago

I suppose it's almost pointless to waste much time on the TinyCore image and revisit this subject after Debian is further along.

My current persistence hack, after starting the vm (but from OSX command line):

Create docker machine boot patch script $ cat ~/.docker/bootsync.sh

#!/bin/sh
sed -i '/if \[ -n \"\$dns\" \]/a     dns=\"$dns 8.8.8.8\"' /usr/share/udhcpc/default.script
#sed -i '/if \[ -n \"\$domain\" \]/a     domain=\"$domain dev\"' /usr/share/udhcpc/default.script

Inject boot script into persistent location on virtual host image $ docker-machine scp ~/.docker/bootsync.sh dev:/mnt/sda1/var/lib/boot2docker/bootsync.sh
Execute script to patch current ephemeral default.script. Or do a docker-machine restart $ docker-machine ssh dev sudo sh /mnt/sda1/var/lib/boot2docker/bootsynch.sh

midwire commented 8 years ago

Confirmed, once I overwrote the dns-crypt proxy settings in resolve.conf within the default VM with nameserver 8.8.8.8 it works.

mchiang0610 commented 8 years ago

Is this still happening on the latest toolbox / Kitematic?

mlushpenko commented 8 years ago

Thanks @mchiang0610 , your suggestion helped me

r4j4h commented 8 years ago

@mchiang0610 your fix is still relevant as of 2/7/1026.

It just helped me on a fresh install of Docker toolbox v1.10.0. My resolv.conf had seemingly valid settings but they gave same results as initial issue reports using just straight docker-machine ssh default and then docker pull commands.

Setting to Google's DNS as you suggest instantly fixed the issue.

Here's what the .conf looked like before the fix, note the first line. I am wondering if it is the root cause:

search ph.cox.net
nameserver 68.105.**.**

FrenchBen commented 8 years ago

@r4j4h Unfortunately not all DNS are made equal - In your case it seems that your ISP's DNS isn't up to par. Using Google's DNS helps quite a bit (probably why they created the service in the first place)

tylerhjones commented 8 years ago

The problem is not the DNS you have, it is the packet return size. Im guessing for 8.8.8.8 the return size is less than 512 bytes. So the UDP packet is forwarded from the vm running on your machine to the host machine. When the DNS returns a packet size over 512 DNS will fall over to TCP, which it looks like the vm is not forwarding to your host machine and thus it cannot resolve.

Notice the "tcp" in "dial tcp: lookup index.docker.io", that indicates to me this is the problem. The "192.168" private address hints to me that this is the private network between your vm and the container.

FrenchBen commented 8 years ago

@tylerhjones thanks for the detailed info. The 192.168 network is created by VirtualBox, which may not forward the TCP packet.

timelf123 commented 7 years ago

The problem is not the DNS you have, it is the packet return size. Im guessing for 8.8.8.8 the return size is less than 512 bytes. So the UDP packet is forwarded from the vm running on your machine to the host machine. When the DNS returns a packet size over 512 DNS will fall over to TCP, which it looks like the vm is not forwarding to your host machine and thus it cannot resolve.

This was it for me - 8.8.8.8 -> 208.67.222.222 (opendns)

deepakdb16 commented 7 years ago

Your docker host may be behind the proxy. Set the proxy in /etc/sysconfig/docker. That should fix the problem.

st0ne-dot-at commented 6 years ago

Hi all, i had similar issues with occasionally occuring lookup outages. I've tracked the issue down to the libc library of the gitea docker image. The image uses the libc.musl-x86_64.so.1 libc library ... which uses an unusual resolve mechanism. It queries the nameservers in parallel (see docker-alpine). This behavior differs from the glibc resolving behavior. The glibc resolver uses always the first entry.

In my case i had a local dns server in the first line, followed by external dns servers. This leaded to resolving errors, because only the local dns server was able to resolve the ip address of the db container.

May that helps.

claudineigomes commented 6 years ago

Thank you so much @mchiang0610 I was trying to "docker pull mongo" and I was getting this message always dial tcp: lookup ...

atlas-comstock commented 5 years ago

@tylerhjones thanks for your explanation.

When the DNS returns a packet size over 512 DNS will fall over to TCP, which it looks like the vm is not forwarding to your host machine and thus it cannot resolve.

Why a packet size over 512 will fall over to TCP?

yhojann-cl commented 4 years ago

Same problem from KVM image using CentOS7

[root@server ~]# docker -v
Docker version 19.03.6, build 369ce74a3c
[root@server ~]# ping -c 1 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=54 time=13.8 ms

--- 8.8.8.8 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 13.823/13.823/13.823/0.000 ms
[root@server ~]# cat /etc/docker/daemon.json
{
    "dns": ["8.8.8.8", "8.8.4.4"]
}
[root@server ~]# docker pull atlassian/jira-software
Using default tag: latest
Trying to pull repository docker.io/atlassian/jira-software ... 
Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 192.168.x.x:53: read udp 192.168.x.x:54328->192.168.x.x:53: i/o timeout
[root@server ~]# DOCKER_OPTS="--dns=8.8.8.8" docker pull atlassian/jira-software
Using default tag: latest
Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 192.168.x.x:53: read udp 192.168.x.x:46200->192.168.x.x:53: i/o timeout
[root@server ~]# dig -t A registry-1.docker.io @192.168.x.x

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-9.P2.el7 <<>> -t A registry-1.docker.io @192.168.x.x
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7747
;; flags: qr rd ra; QUERY: 1, ANSWER: 8, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;registry-1.docker.io.      IN  A

;; ANSWER SECTION:
registry-1.docker.io.   42  IN  A   34.199.40.84
registry-1.docker.io.   42  IN  A   52.202.253.107
registry-1.docker.io.   42  IN  A   34.201.196.144
registry-1.docker.io.   42  IN  A   34.205.88.205
registry-1.docker.io.   42  IN  A   34.228.211.243
registry-1.docker.io.   42  IN  A   34.232.31.24
registry-1.docker.io.   42  IN  A   52.201.142.14
registry-1.docker.io.   42  IN  A   34.199.77.19

;; Query time: 0 msec
;; SERVER: 192.168.x.x#53(192.168.x.x)
;; WHEN: sáb feb 15 23:49:13 -03 2020
;; MSG SIZE  rcvd: 177

There is something curious to see the status of the firewalld service and it is that the docker rules commands have failed since installation. Is the connection to the DNS service due to a problem in the firewall rules applied automatically in the docker installation?

[root@server ~]# cat /var/log/firewalld
2020-02-15 23:04:43 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER' failed: iptables v1.4.21: Couldn't load target `DOCKER':No such file or directory
Try `iptables -h' or 'iptables --help' for more information.
2020-02-15 23:04:43 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables v1.4.21: Couldn't load target `DOCKER':No such file or directory
Try `iptables -h' or 'iptables --help' for more information.
2020-02-15 23:04:43 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D PREROUTING' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2020-02-15 23:04:43 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D OUTPUT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2020-02-15 23:04:43 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -F DOCKER' failed: iptables: No chain/target/match by that name.
2020-02-15 23:04:43 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -X DOCKER' failed: iptables: No chain/target/match by that name.
2020-02-15 23:04:43 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER' failed: iptables: No chain/target/match by that name.
2020-02-15 23:04:43 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER' failed: iptables: No chain/target/match by that name.
2020-02-15 23:04:43 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name.
2020-02-15 23:04:43 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name.
2020-02-15 23:04:43 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2020-02-15 23:19:01 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables: No chain/target/match by that name.
2020-02-15 23:19:01 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D PREROUTING' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2020-02-15 23:19:01 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D OUTPUT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2020-02-15 23:19:01 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER' failed: iptables: Too many links.
2020-02-15 23:19:01 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATION' failed: iptables: Too many links.
2020-02-15 23:19:01 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2020-02-15 23:19:01 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2020-02-15 23:33:57 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables: No chain/target/match by that name.
2020-02-15 23:33:57 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D PREROUTING' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2020-02-15 23:33:57 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D OUTPUT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2020-02-15 23:33:57 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER' failed: iptables: Too many links.
2020-02-15 23:33:57 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER-ISOLATION-STAGE-1' failed: iptables: No chain/target/match by that name.
2020-02-15 23:33:57 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATION-STAGE-1' failed: iptables: No chain/target/match by that name.
2020-02-15 23:33:57 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER-ISOLATION-STAGE-2' failed: iptables: No chain/target/match by that name.
2020-02-15 23:33:57 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATION-STAGE-2' failed: iptables: No chain/target/match by that name.
2020-02-15 23:33:57 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2020-02-15 23:33:57 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2020-02-15 23:35:17 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables: No chain/target/match by that name.
2020-02-15 23:35:17 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D PREROUTING' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2020-02-15 23:35:17 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D OUTPUT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2020-02-15 23:35:17 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER' failed: iptables: Too many links.
2020-02-15 23:35:17 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATION-STAGE-1' failed: iptables: Too many links.
2020-02-15 23:35:17 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name.
2020-02-15 23:35:17 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name.
2020-02-15 23:35:17 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2020-02-15 23:35:18 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2020-02-16 00:04:41 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2020-02-16 00:04:41 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2020-02-16 00:08:03 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2020-02-16 00:08:03 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).

I cannot modify the DNS server globally on my server because I use a DNS service with custom internal domains, although my DNS service still works well, it is KVM.

Everything indicates that there must be a bug in the docker DNS query connector and not the system or system DNS configuration.

Temporaly solution is add the address to /etc/hosts:

34.228.211.243  registry-1.docker.io
34.205.88.205   auth.docker.io
104.18.121.25   production.cloudflare.docker.com

But i lost the DNS balancing and address integrity for changes.

docker / kitematic

DNS fails on image pull: dial tcp: lookup index.docker.io #718