failed to copy: httpReadSeeker: failed open: failed to do request: Get "https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/24/24c63b8dcb66721062f32b893ef1027404afddd62aade87f3f39a3a6e70a74d0/data?verify=1717211225-SoFsY9MpCnMY8xiypN2ii7WhLsA%3D": EOF

[nico-arianto]

Description

This issue only happen when use containerd but it's fine when use the default image store

Reproduce

docker image pull <new image that not available in local>

Expected behavior

docker image pull <new image that not available in local> should be able to pull the image successfully

docker version

Client:
 Cloud integration: v1.0.35+desktop.13
 Version:           26.1.1
 API version:       1.45
 Go version:        go1.21.9
 Git commit:        4cf5afa
 Built:             Tue Apr 30 11:44:56 2024
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.30.0 (149282)
 Engine:
  Version:          26.1.1
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.9
  Git commit:       ac2de55
  Built:            Tue Apr 30 11:48:04 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.31
  GitCommit:        e377cd56a71523140ca6ae87e30244719194a521
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    26.1.1
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.14.0-desktop.1
    Path:     /Users/nicoarianto/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.27.0-desktop.2
    Path:     /Users/nicoarianto/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.29
    Path:     /Users/nicoarianto/.docker/cli-plugins/docker-debug
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/nicoarianto/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.23
    Path:     /Users/nicoarianto/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.4
    Path:     /Users/nicoarianto/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.1.0
    Path:     /Users/nicoarianto/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/nicoarianto/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.8.0
    Path:     /Users/nicoarianto/.docker/cli-plugins/docker-scout

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 26.1.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e377cd56a71523140ca6ae87e30244719194a521
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.6.26-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 10
 Total Memory: 7.657GiB
 Name: docker-desktop
 ID: 3ea9f31b-c086-4f80-9c27-7d97d9194a4c
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/nicoarianto/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

Diagnostics ID

D44C6E40-9F9F-4971-A1D6-6BDD6C7E6DC5/20240601022618

Additional Info

No response

[nicks]

to anyone experiencing this issue: could you post the output of a DNS trace? e.g.

dig +trace production.cloudflare.docker.com

[8i5dev]

$ dig +trace production.cloudflare.docker.com

; <<>> DiG 9.16.1-Ubuntu <<>> +trace production.cloudflare.docker.com ;; global options: +cmd . 0 IN NS k.root-servers.net. . 0 IN NS f.root-servers.net. . 0 IN NS a.root-servers.net. . 0 IN NS d.root-servers.net. . 0 IN NS l.root-servers.net. . 0 IN NS g.root-servers.net. . 0 IN NS i.root-servers.net. . 0 IN NS e.root-servers.net. . 0 IN NS m.root-servers.net. . 0 IN NS c.root-servers.net. . 0 IN NS j.root-servers.net. . 0 IN NS b.root-servers.net. . 0 IN NS h.root-servers.net. ;; Received 432 bytes from 172.17.32.1#53(172.17.32.1) in 0 ms

com. 172800 IN NS a.gtld-servers.net. com. 172800 IN NS b.gtld-servers.net. com. 172800 IN NS c.gtld-servers.net. com. 172800 IN NS d.gtld-servers.net. com. 172800 IN NS e.gtld-servers.net. com. 172800 IN NS f.gtld-servers.net. com. 172800 IN NS g.gtld-servers.net. com. 172800 IN NS h.gtld-servers.net. com. 172800 IN NS i.gtld-servers.net. com. 172800 IN NS j.gtld-servers.net. com. 172800 IN NS k.gtld-servers.net. com. 172800 IN NS l.gtld-servers.net. com. 172800 IN NS m.gtld-servers.net. com. 86400 IN DS 19718 13 2 8ACBB0CD28F41250A80A491389424D341522D946B0DA0C0291F2D3D7 71D7805A com. 86400 IN RRSIG DS 8 1 86400 20240619170000 20240606160000 5613 . I/XRBWN3F3pXhkJyRlERZeFzc4rTxeWgsiCUD74hwmtte/kXbLqy6Mjj RTVBb/hoJ1lH9eVZby4Xfy/Blu85ik99bDXQxGpykZKYBmqTKTlxekEV +Eu4XNFqXoQwbXIW9eh0KjqZkPnMNsaJcK/ztj8/yKwVTZnv19I8NYxO x+ca/SgONadkQExvZ6xbp5/j6HFEDamwVW7D/8cTD0jdHXFMapgzDnh5 uMKT/AK3PJW7oYIk5jRxEiJ6TPJfIB5Hcwmv9L32SQbtmh3ukkDUTE9n GdurV4Ok3eZNk9Z0qCdQDLIWlsEJgCDevBDC/Xq7TBQtoREIZocoqSno fHLfqA== ;; Received 1192 bytes from 199.7.83.42#53(l.root-servers.net) in 20 ms

production.cloudflare.docker.com. 181 IN A 74.86.151.167 ;; Received 77 bytes from 192.35.51.30#53(f.gtld-servers.net) in 0 ms

[delai]

same issue in ubuntu 22.04 since yestoday

[cookieY]

same issue in debian since yestoday

[liuchenbaidu]

same issue in ubuntu

[ColdsteelRail]

same issue on mac os

[nico-arianto]

Here is my DNS trace

❯ dig +trace production.cloudflare.docker.com

; <<>> DiG 9.10.6 <<>> +trace production.cloudflare.docker.com
;; global options: +cmd
.           0   IN  NS  h.root-servers.net.
.           0   IN  NS  l.root-servers.net.
.           0   IN  NS  d.root-servers.net.
.           0   IN  NS  f.root-servers.net.
.           0   IN  NS  i.root-servers.net.
.           0   IN  NS  b.root-servers.net.
.           0   IN  NS  g.root-servers.net.
.           0   IN  NS  c.root-servers.net.
.           0   IN  NS  k.root-servers.net.
.           0   IN  NS  a.root-servers.net.
.           0   IN  NS  j.root-servers.net.
.           0   IN  NS  e.root-servers.net.
.           0   IN  NS  m.root-servers.net.
.           0   IN  RRSIG   NS 8 0 518400 20240620050000 20240607040000 5613 . YcPStR37RG8953PyZUXUfW63L9sPLZbMiGCywWQ1dwOSKp8foAyQEGOJ bzacKj49OV4PmwgkdP6w2STwEHmTrbX/w/+s1w5LBx5M2UhaC3Ad7JhG 37vaEZjCxA/nYRfQz+wQ0rN6v4RAp/qMzKrURQ+XGICnt68vZyFeC6qW tBVeAvnBtY1aoZMJ3Ab5mTx99AZ9jlUZeuyGeVhsCM+onrZF8J8bTPjd EdPiotUHJ4oSh3yv4LiUOvSLz7bqUHMjaxTfb5Z+/43lVHIe55xmAM2F s9tobDsJQXTexKaISEyGrnaP0moR5ZRfiYompFMJfkpa1Kco+LozYUsw jy9C5w==
;; Received 541 bytes from 10.0.0.1#53(10.0.0.1) in 49 ms

com.            172800  IN  NS  h.gtld-servers.net.
com.            172800  IN  NS  b.gtld-servers.net.
com.            172800  IN  NS  l.gtld-servers.net.
com.            172800  IN  NS  d.gtld-servers.net.
com.            172800  IN  NS  k.gtld-servers.net.
com.            172800  IN  NS  j.gtld-servers.net.
com.            172800  IN  NS  g.gtld-servers.net.
com.            172800  IN  NS  a.gtld-servers.net.
com.            172800  IN  NS  i.gtld-servers.net.
com.            172800  IN  NS  f.gtld-servers.net.
com.            172800  IN  NS  e.gtld-servers.net.
com.            172800  IN  NS  c.gtld-servers.net.
com.            172800  IN  NS  m.gtld-servers.net.
com.            86400   IN  DS  19718 13 2 8ACBB0CD28F41250A80A491389424D341522D946B0DA0C0291F2D3D7 71D7805A
com.            86400   IN  RRSIG   DS 8 1 86400 20240620050000 20240607040000 5613 . IScEf6/d4yPrc5CrcYzL74L30Ue4Xh3EbgMCIW32uFstGZJYKCLUEXzN tqpptEWjmnJ0cLMtNjxiHEL/hntdooRO59+xajU1/XNk2sIHKLOKy9MF XIQGsVQqJ5NTQjtT81zQ2+brcC6V86oBnrBHDzJMJXa4mWyBr5FmwoJW HVKOmatMNcyxD5L7r7+NF3OgVHb6EWXt98lX02z9HYFmeSc6JL3UtJix X3gXCpGa5F/l4T1sR4tzy8QdC9qObAuTr7iEw9vyvsVxxZ58/kSS5yU3 1iQhSELReq4YinF8TycdEwok0M0oDoX9nZKCBYyO5Ozkoma+QfQIW4Fu wwSJuA==
;; Received 1195 bytes from 202.12.27.33#53(m.root-servers.net) in 162 ms

docker.com.     172800  IN  NS  ns-207.awsdns-25.com.
docker.com.     172800  IN  NS  ns-568.awsdns-07.net.
docker.com.     172800  IN  NS  ns-1289.awsdns-33.org.
docker.com.     172800  IN  NS  ns-1981.awsdns-55.co.uk.
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN NSEC3 1 1 0 - CK0Q2D6NI4I7EQH8NA30NS61O48UL8G5  NS SOA RRSIG DNSKEY NSEC3PARAM
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN RRSIG NSEC3 13 2 86400 20240613042504 20240606031504 956 com. P2CEliINyDTK2VfAZVsAIDYjkicDEIgbOAJQ0LS7BcmNG0LogEpxx6zq /tG7ddSvTm//eHhTegKNG6jFZlEGpA==
8CELDSU2MN5PAAUQM465IP3UJ7AAEFTR.com. 86400 IN NSEC3 1 1 0 - 8CELJRVJR4EPPS1K7AB5Q6NJPDRRRMRQ  NS DS RRSIG
8CELDSU2MN5PAAUQM465IP3UJ7AAEFTR.com. 86400 IN RRSIG NSEC3 13 2 86400 20240614053706 20240607042706 956 com. LB2IrRTCf+iY1dk5zZNIsbQtVSHeqh4Ft22eGfEHY9XNndoo4h1YAe14 JkoVPtMeZno0Gcuu7cUibS63iIMIjw==
;; Received 571 bytes from 192.35.51.30#53(f.gtld-servers.net) in 46 ms

production.cloudflare.docker.com. 300 IN NS dara.ns.cloudflare.com.
production.cloudflare.docker.com. 300 IN NS rob.ns.cloudflare.com.
;; Received 112 bytes from 205.251.199.189#53(ns-1981.awsdns-55.co.uk) in 21 ms

production.cloudflare.docker.com. 300 IN A  104.16.97.215
production.cloudflare.docker.com. 300 IN A  104.16.99.215
production.cloudflare.docker.com. 300 IN A  104.16.100.215
production.cloudflare.docker.com. 300 IN A  104.16.98.215
production.cloudflare.docker.com. 300 IN A  104.16.101.215
;; Received 141 bytes from 108.162.192.91#53(dara.ns.cloudflare.com) in 56 ms

[darkyojimbo]

Here is the error that I am getting, not sure if this is same or not:

xxxx@xxx:~/01. email project $ docker build -t email-test .
[+] Building 34.0s (3/3) FINISHED                                                                                                                                                              docker:default
 => [internal] load build definition from dockerfile                                                                                                                                                     0.0s
 => => transferring dockerfile: 829B                                                                                                                                                                     0.0s
 => [internal] load .dockerignore                                                                                                                                                                        0.0s
 => => transferring context: 2B                                                                                                                                                                          0.0s
 => ERROR [internal] load metadata for docker.io/library/python:3.10-slim                                                                                                                               33.7s
------
 > [internal] load metadata for docker.io/library/python:3.10-slim:
------
dockerfile:2
--------------------
   1 |     # Use the official Python image from the Docker Hub
   2 | >>> FROM python:3.10-slim
   3 |
   4 |     # Install cron
--------------------
ERROR: failed to solve: DeadlineExceeded: DeadlineExceeded: DeadlineExceeded: python:3.10-slim: failed to copy: httpReadSeeker: failed open: failed to do request: Get "https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/8d/8de37b2aed5bbbc215fe4720179a41de6567a849dbbbec53286b8ff7cbd3e648/data?verify=1717950288-5zeknQUEYvIObPdY6COEtszZzMI%3D": dial tcp 128.242.245.180:443: i/o timeout

And here is my OS:

xxx@xx:~/01. email project $ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 11 (bullseye)
Release:        11
Codename:       bullseye

Here is my result to digcommand:

xxx@xx:~/01. email project $ dig +trace production.cloudflare.docker.com

; <<>> DiG 9.16.48-Debian <<>> +trace production.cloudflare.docker.com
;; global options: +cmd
;; connection timed out; no servers could be reached

Can anyone advice please?

[kevinyx2]

has anyone solved this? faced this issue yesterday

[igotcha]

same issue:

$ dig +trace production.cloudflare.docker.com

; <<>> DiG 9.10.6 <<>> +trace production.cloudflare.docker.com
;; global options: +cmd
.                       788     IN      NS      l.root-servers.net.
.                       788     IN      NS      e.root-servers.net.
.                       788     IN      NS      k.root-servers.net.
.                       788     IN      NS      c.root-servers.net.
.                       788     IN      NS      i.root-servers.net.
.                       788     IN      NS      m.root-servers.net.
.                       788     IN      NS      f.root-servers.net.
.                       788     IN      NS      j.root-servers.net.
.                       788     IN      NS      h.root-servers.net.
.                       788     IN      NS      d.root-servers.net.
.                       788     IN      NS      a.root-servers.net.
.                       788     IN      NS      g.root-servers.net.
.                       788     IN      NS      b.root-servers.net.
;; Received 431 bytes from 10.251.1.1#53(10.251.1.1) in 2022 ms

production.cloudflare.docker.com. 208 IN A      104.23.124.189
;; Received 77 bytes from 198.97.190.53#53(h.root-servers.net) in 23 ms

[wangjunji]

same issue here. still not recovered

[samzong]

same issue.

samzonglu in ~/Desktop λ dig +trace production.cloudflare.docker.com

; <<>> DiG 9.10.6 <<>> +trace production.cloudflare.docker.com
;; global options: +cmd
.           1799    IN  NS  l.root-servers.net.
.           1799    IN  NS  g.root-servers.net.
.           1799    IN  NS  f.root-servers.net.
.           1799    IN  NS  k.root-servers.net.
.           1799    IN  NS  b.root-servers.net.
.           1799    IN  NS  c.root-servers.net.
.           1799    IN  NS  e.root-servers.net.
.           1799    IN  NS  j.root-servers.net.
.           1799    IN  NS  i.root-servers.net.
.           1799    IN  NS  m.root-servers.net.
.           1799    IN  NS  h.root-servers.net.
.           1799    IN  NS  d.root-servers.net.
.           1799    IN  NS  a.root-servers.net.
;; Received 239 bytes from 10.64.46.101#53(10.64.46.101) in 5 ms

production.cloudflare.docker.com. 104 IN A  104.244.43.104
;; Received 77 bytes from 198.97.190.53#53(h.root-servers.net) in 8 ms

[zchunhai]

same issue on mac os

[mailliw2010]

same issue on ubuntu 22.04 when used containerd as image endpoint:

export CONTAINER_RUNTIME_ENDPOINT=unix:///run/containerd/containerd.sock crictl pull docker.io/calico/cni:v3.26.1

FATA[0028] pulling image: rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/calico/cni:v3.26.1": failed to copy: httpReadSeeker: failed open: failed to do request: Get "https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/9d/9dee260ef7f5990aaf6e8f6767b767366c27a6abbf79ba8dba45ff3290bd5de0/data?verify=1718595102-BSzxBoXFPcWt1zQBxZL0v9TjauI%3D": dial tcp 75.126.124.162:443: connect: connection refused

[kevinyx2]

apparently, it seems docker is blocked in china, so i resolved this issue by manually pulling the docker base image from a local VM or anywhere that can access cloudfare/docker....and then manually deploy the base image on prem...and it works

[yxw007]

My solution is to modify the dns and host so that ping production.cloudflare.docker.com can be pinged.

[gjhuai]

use docker proxy

[NathanChan]

My solution is to modify the dns and host so that ping production.cloudflare.docker.com can be pinged. Adding a host alias work for me too, see this reply: https://github.com/docker/hub-feedback/issues/2388#issuecomment-2173931387

[LeviPesin]

Same issue on Windows. WSL log:

$ dig +trace production.cloudflare.docker.com
;; communications error to 10.255.255.254#53: timed out
;; communications error to 10.255.255.254#53: timed out
;; communications error to 10.255.255.254#53: timed out

; <<>> DiG 9.18.18-0ubuntu0.22.04.1-Ubuntu <<>> +trace production.cloudflare.docker.com
;; global options: +cmd
;; no servers could be reached

[NathanChan]

邮件已收到！辛苦了，我会在第一时间回复您的！

[LeviPesin]

When opening the domain in browser I get just:

{
    "status": 403,
    "message": "Error: invalid URL signature"
}

[LeviPesin]

Now the extensions seems to be installing again, seems like a temporary (~15 minutes) outage.