Docker Pull Timeout

[Supermarcel10]

Description

Unable to pull any project to a docker container. Running a docker-compose pull command always results in a "request canceled (Client.Timeout exceeded while awaiting headers)" error.

I have attempted restarting docker. I have attempted logging in to docker again - but this caused another error:

Error response from daemon: Get "https://registry-1.docker.io/v2/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

I have attempted to set the DNS with /etc/resolve.confg to 8.8.8.8/8.8.4.4

Steps to reproduce the issue:

1. Run docker pull hello-world in console.
2. Wait for the command to execute.

Describe the results you received: Output:

Using default tag: latest
Error response from daemon: Head "https://registry-1.docker.io/v2/library/hello-world/manifests/latest": Get "https://auth.docker.io/token?scope=repository%3Alibrary%2Fhello-world%3Apull&service=registry.docker.io": net/http: request canceled (Client.Timeout exceeded while awaiting headers)

Describe the results you expected: A successfull pull of hello-world

Additional information you deem important (e.g. issue happens only occasionally): Attempted multiple things to fix this error, but it still seems to persist. I have also asked on the official docker discord, but all the links and tutorials I have tried that were given to me still caused this error.

Output of docker compose version:

version 1.29.2, build unknown

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.8.1-docker)
  scan: Docker Scan (Docker Inc., v0.17.0)

Server:
 Containers: 14
  Running: 2
  Paused: 0
  Stopped: 12
 Images: 47
 Server Version: 20.10.14
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux       
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3df54a852345ae127d1fa3092b95168e4a88e2f8
 runc version: v1.0.3-0-gf46b6ba
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.4.0-104-generic
 Operating System: Ubuntu 20.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 6
 Total Memory: 19.55GiB
 Name: sr2-kvm
 ID: YXRA:BBLB:HW3K:EBFL:5MCK:DZLO:L6QJ:UIHI:R6ZC:AS5O:MN7M:62O5
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details:

[ndeloof]

You mentioned docker pull hello-world for reproduction, is this issue related to compose or plain docker cli?

[Supermarcel10]

It seems like a docker compose issue to me, since I have other things in a docker running, but I'm unable to compose new docker containers.

[ndeloof]

"Steps to reproduce the issue" you mentioned do not involve any docker compose command, not compose.yaml file, so I hardly understand what this is about and how to help you / investigate for a bug

[Supermarcel10]

The original issue was with docker-compose, but it seems to span both docker and docker-compose commands. When using docker-compose build to build cabot web and worker services it errors out. After testing and attempting to use common fixed I have determined that both docker and docker-compose time out.

By following this tutorial from the official cabot website (https://cabotapp.com/dev/get-started.html):

# docker-compose build
redis uses an image, skipping
db uses an image, skipping
Building beat
Step 1/16 : FROM node:4-alpine
ERROR: Service 'beat' failed to build: Head "https://registry-1.docker.io/v2/library/node/manifests/4-alpine": Get "https://auth.docker.io/token?scope=repository%3Alibrary%2Fnode%3Apull&service=registry.docker.io": net/http: request canceled (Client.Timeout exceeded while awaiting headers)

[CarlosLandin2]

This worked for me

# edit your DNS file

vi /etc/resolv.conf

comment the following line:

nameserver 127.0.0.1

and add the following line:

nameserver 8.8.8.8

[aldobranti]

Every place else there is a claim that changing resolv behavior works and suggest picking up google.com name servers here is output of dig on registry.docker.io, so you can see that I am using 8.8.8.8

`dig registry.docker.io

; <<>> DiG 9.16.27-Debian <<>> registry.docker.io

;; global options: +cmd

;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11419

;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:

; EDNS: version: 0, flags:; udp: 512

;; QUESTION SECTION:

;registry.docker.io. IN A

;; ANSWER SECTION:

registry.docker.io. 13 IN CNAME registry-1.docker.io.

registry-1.docker.io. 30 IN A 44.207.51.64

registry-1.docker.io. 30 IN A 44.207.96.114

registry-1.docker.io. 30 IN A 44.194.5.25

;; Query time: 11 msec

;; SERVER: 8.8.8.8#53(8.8.8.8)

;; WHEN: Mon Jul 04 09:27:39 BST 2022

;; MSG SIZE rcvd: 120`

I get a whole spectrum of error messages on pulls and run

`Error response from daemon: Head "https://registry-1.docker.io/v2/kasmweb/gimp/manifests/latest": dial tcp: i/o timeout

Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Error response from daemon: Head "https://registry-1.docker.io/v2/library/hello-world/manifests/arm64": Get "https://auth.docker.io/token?scope=repository%3Alibrary%2Fhello-world%3Apull&service=registry.docker.io": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers). ` Got to ask, if I splash out $60 can I at least expect to get a simple pull to work -- is this a reflection that impoverished artists [ read cheapskate ] just have to put up with this ?

[lockejan]

In my case it has been a network issue of my vps provider. No need to alter resolv.conf. Have you tried it again after a few days?

[aldobranti]

I simply wrote a script to continue repeatedly attempting the download  , it generally gets there -- aldobranti.org, a performance  -------- Original message --------From: Jan Schmitt @.> Date: 01/10/2022 13:48 (GMT+00:00) To: docker/compose @.> Cc: aldobranti @.>, Comment @.> Subject: Re: [docker/compose] Docker Pull Timeout (Issue #9360) In my case it has been a network issue of my vps provider. No need to alter resolv.conf. Have you tried it again after a few days?

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

[kjkent]

I am also having this issue, specifically with Docker Compose. While changing /etc/resolv.conf to use Google DNS does solve the issue on the host, it's not an ideal long term solution as I'd prefer all DNS to be routed through the local server.

FWIW, if on my local server I set the firewall to port forward any WAN-bound DNS queries to the local Unbound resolver, this breaks Docker Compose, regardless of the host's resolv.conf setup.

Interestingly, I don't encounter this issue with my personal machine running Fedora 36 Workstation, but I do on my LAN server, which is running a minimal install of Debian 11.6. Perhaps the biggest difference in TCP/IP stack is that the Debian machine is using it's inbuilt networking service (not systemd-networkd or NetworkManager), whereas my personal one is using NetworkManager.

Edit: If there's anything I can do to help debug, please let me know.

[ndeloof]

@kjkent can you please check you get docker compose pull to fail, but can pull individual images with docker pull?

[kjkent]

@ndeloof I just checked this, and you're right -- pulling an individual image (using docker compose pull jellyfin, for instance) succeeds, whereas puling the whole 10-container compose-file fails with i/o timeout

[Bug-Reaper]

This worked for me

# edit your DNS file

vi /etc/resolv.conf

comment the following line:

nameserver 127.0.0.1

and add the following line:

nameserver 8.8.8.8

Thank you so much! This worked for me!

[ndeloof]

I'm closing this issue as obsolete: Docker Compose V1 has reached end-of-life and we are not accepting any more changes. Please try and reproduce your issue with Compose V2 and create a new issue or PR with the relevant Compose V2 information.