search

docker / for-linux

Docker Engine for Linux
https://docs.docker.com/engine/installation/
753 stars 85 forks source link

Docker ignoring DNS? #257

Open mrots opened 6 years ago

mrots commented 6 years ago

Expected behavior

docker pull <image> - pulls the image docker search <image> - shows search results docker login - logs in

Actual behavior

Client.Timeout exceeded while awaiting headers

Steps to reproduce the behavior

$ docker run hello-world
Unable to find image 'hello-world:latest' locally
Trying to pull repository docker.io/library/hello-world ... 
/usr/bin/docker-current: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers).
See '/usr/bin/docker-current run --help'.
$ docker pull whalesay
Using default tag: latest
Trying to pull repository docker.io/library/whalesay ... 
Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Packet capture shows the host querying public resolvers (OpenDNS) successfully. No sensitive data contained below, this is a sandbox environment using RFC1918 address space.

10:53:13.955187 IP 192.168.200.11.50854 > 208.67.222.222.53: 25976+ AAAA? registry-1.docker.io. (38)
10:53:13.955187 IP 192.168.200.11.37188 > 208.67.222.222.53: 17049+ A? registry-1.docker.io. (38)
10:53:13.973254 IP 208.67.222.222.53 > 192.168.200.11.50854: 25976 0/13/10 (409)
10:53:13.977060 IP 208.67.222.222.53 > 192.168.200.11.37188: 17049 8/13/10 A 52.20.146.203, A 52.22.181.254, A 34.200.28.105, A 54.152.209.167, A 52.204.202.231, A 35.169.231.249, A 52.5.185.86, A 34.200.90.16 (537)
10:53:18.955286 IP 192.168.200.11.53976 > 208.67.220.220.53: 2522+ A? registry-1.docker.io. (38)
10:53:18.958577 IP 208.67.220.220.53 > 192.168.200.11.53976: 2522 8/13/10 A 52.22.181.254, A 34.200.28.105, A 54.152.209.167, A 52.204.202.231, A 35.169.231.249, A 52.5.185.86, A 34.200.90.16, A 52.20.146.203 (537)
10:53:23.955388 IP 192.168.200.11.41320 > 208.67.222.222.53: 53562+ A? registry-1.docker.io. (38)
10:53:23.958729 IP 208.67.222.222.53 > 192.168.200.11.41320: 53562 8/13/10 A 34.200.28.105, A 54.152.209.167, A 52.204.202.231, A 35.169.231.249, A 52.5.185.86, A 34.200.90.16, A 52.20.146.203, A 52.22.181.254 (537)
10:53:28.955530 IP 192.168.200.11.56829 > 208.67.220.220.53: 25610+ A? registry-1.docker.io. (38)
10:53:28.983831 IP 208.67.220.220.53 > 192.168.200.11.56829: 25610 8/13/10 A 52.204.202.231, A 35.169.231.249, A 52.5.185.86, A 52.20.146.203, A 52.22.181.254, A 34.200.28.105, A 54.152.209.167, A 54.164.230.151 (537)

No TCP/443 connection initiation attempts to registry-1.docker.io were observed, suggesting that the DNS query results aren't being received by the Docker application itself.

Able to curl registry-1.docker.io and get a response without issue.

$ curl https://registry-1.docker.io/v2/
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}

Output of docker version:

$ docker version
Client:
 Version:         1.13.1
 API version:     1.26
 Package version: <unknown>
 Go version:      go1.8.3
 Git commit:      774336d/1.13.1
 Built:           Wed Mar  7 17:06:16 2018
 OS/Arch:         linux/amd64

Server:
 Version:         1.13.1
 API version:     1.26 (minimum version 1.12)
 Package version: <unknown>
 Go version:      go1.8.3
 Git commit:      774336d/1.13.1
 Built:           Wed Mar  7 17:06:16 2018
 OS/Arch:         linux/amd64
 Experimental:    false

Output of docker info:

$ docker info
Containers: 4
 Running: 2
 Paused: 0
 Stopped: 2
Images: 12
Server Version: 1.13.1
Storage Driver: devicemapper
 Pool Name: docker-253:0-718804-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 1.841 GB
 Data Space Total: 107.4 GB
 Data Space Available: 95.62 GB
 Metadata Space Used: 4.067 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.143 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.140-RHEL7 (2017-05-03)
Logging Driver: journald
Cgroup Driver: systemd
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Init Binary: docker-init
containerd version:  (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: N/A (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: N/A (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 seccomp
  WARNING: You're not using the default seccomp profile
  Profile: /etc/docker/seccomp.json
Kernel Version: 3.10.0-693.21.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 4
Total Memory: 7.638 GiB
Name: neptune
ID: 55JI:RPCG:N6DQ:R6YV:2R23:PUWH:A64D:AJD7:WBRH:P32K:4EME:6333
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Registries: docker.io (secure)

Additional environment details (AWS, VirtualBox, physical, etc.) This is a physical CentOS 7 server with all the latest yum updates. Networking is standard layer 3 routing, no proxies involved.

This issue seems to circle around the notion that Docker is possibly ignoring the DNS responses being received by the host. Any direction or suggestion would be much appreciated!

mrots commented 6 years ago

A bit more food for thought...