search

docker / hub-feedback

Feedback and bug reports for the Docker Hub
https://hub.docker.com
233 stars 48 forks source link

Access to hub is blocked, can you remove 149.86.61.9 from the blacklist - "connection reset by peer" #2271

Closed projx closed 1 year ago

projx commented 1 year ago

Hi

My ISP issued a new IP to my router a few days ago, and every since I've been unable to pull or push to Docker hub, I get the following results every time:

root@dcker-01:~/netmaker# docker pull gravitl/netmaker:v0.16.0 Error response from daemon: Get "https://registry-1.docker.io/v2/": read tcp 10.10.40.44:35514->3.216.34.172:443: read: connection reset by peer

I have also tried using different DNS server, and even tested each of the the specific IPs returned by the DNS response:

Non-authoritative answer: Name: registry-1.docker.io Address: 34.205.13.154 Name: registry-1.docker.io Address: 3.216.34.172 Name: registry-1.docker.io Address: 44.205.64.79

Testing them individually by adding them to /etc/hosts, but I get the same result. So in order to do any work, I'm currently having to route via a VPN.

Thanks

ingshtrom commented 1 year ago

This is strange. We cannot look up your IP address since it only shows your internal IP on the error logs. If you did get blocked by us, you will receive a full HTTP response in return, so we are confident we are not blocking your public IP address.

We believe this is some network issue between you and us--I would probably start with your ISP since they clearly made some changes (maybe it was a normal IP rotation, maybe something more). It is even more strange that using a VPN works around this problem unless it truly is something at the ISP layer that is breaking/blocking you.

Have you tried using traceroute to attempt to figure out where in the network path the issue is?

projx commented 1 year ago

Hi

Doh! that was a rookie mistake, sorry the IP is 149.86.61.9

The reason I think its its the docker hub side, if I visit https://registry-1.docker.io/v2/ in a browser I do get the following message:

{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}

I only get the connection "reset by peer" when using the docker client, also I get the same issue when pulling images to my K3S cluster - yet, if I route over either my docker host, or K3S cluster via a VPN, the images pull fine.

I have also checked on my home Firewall, and I can't see this being done by an interim network that I'm routing over, as to only block calls for the image pull, they'd need to be able to inspect the traffic, but both browser and docker-client traffic is TLS encrypted.

Thanks

ingshtrom commented 1 year ago

👋 Hiya,

Interesting, this is odd indeed. I checked that your IP is not being rate-limited, even though you would still receive a full response if you were.

So here are the requests you have made today (based on UTC) c730e60a-72d5-4e8d-a687-6a25fe753429.csv. It just seems like the Docker CLI is failing to do anything except check if auth is already set 🤔

docker-pull-in-bash could also be enlightening to know which requests are being attempted and failing. You can run it like ./docker_pull.sh gravitl/netmaker v0.16.0 registry-1.docker.io. It should automatically remove the Authorization headers in the logs, but make sure to double check!

The other thing that could be happening, is there is a proxy in the middle somewhere which is blocking this. It is very odd how you seem to be able to hit our hosts, but then subsequent requests fail 🤔

In the meantime, I will ask internally to see if there is something else that could be amiss.

projx commented 1 year ago

Thanks, I'll give docker-pull-in-bash a try.. This is a home broadband connection, so there is no proxy, unless its somewhere upstream, but its of little use, as traffic is encrypted and my VMs would not trust any attempt to MITM it.

projx commented 1 year ago

Same result with docker-pull-in-bash.

The output when routing normalling is:

   Copyright (C) 2021 Free Software Foundation, Inc.
   License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
   This is free software: you are free to change and redistribute it.
   There is NO WARRANTY, to the extent permitted by law.

   Written by Mike Haertel and others; see
   <https://git.sv.gnu.org/cgit/grep.git/tree/AUTHORS>.
   ----- jq version -----jq-1.6
   ----- sed version -----
   sed (GNU sed) 4.8
   Packaged by Debian
   Copyright (C) 2020 Free Software Foundation, Inc.
   License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
   This is free software: you are free to change and redistribute it.
   There is NO WARRANTY, to the extent permitted by law.

   Written by Jay Fenlason, Tom Lord, Ken Pizzini,
   Paolo Bonzini, Jim Meyering, and Assaf Gordon.

   This sed program was built with SELinux support.
   SELinux is disabled on this system.

   GNU sed home page: <https://www.gnu.org/software/sed/>.
   General help using GNU software: <https://www.gnu.org/gethelp/>.
   E-mail bug reports to: <bug-sed@gnu.org>.
   ----- curl version -----
   curl 7.81.0 (x86_64-pc-linux-gnu) libcurl/7.81.0 OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.13
   Release-Date: 2022-01-05
   Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
   Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets zstd

   ----- is this a sha instead of a tag? false  -----
   ----- pulling image registry-1.docker.io/gravitl/netmaker:v0.16.0 -----
   curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to auth.docker.io:443
   ----- digest:  -----
   curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to registry-1.docker.io:443
   ----- new digest:  -----
   curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to registry-1.docker.io:443
   ----- final manifest:  -----

When I try to tellnet to the auth and registry-1 on 443, I seem to get a response, but then it closes: 

   root@dcker-01:~/t/docker-pull-in-bash# telnet auth.docker.io 443
   Trying 3.216.34.172...
   Connected to auth.docker.io.
   Escape character is '^]'.
   Connection closed by foreign host.

When I'm connected via VPN, it pulls the image fine:

   root@dcker-01:~/t/docker-pull-in-bash# ./docker_pull.sh gravitl/netmaker v0.16.0 registry-1.docker.io
   ----- grep version -----grep (GNU grep) 3.7
   Copyright (C) 2021 Free Software Foundation, Inc.
   License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
   This is free software: you are free to change and redistribute it.
   There is NO WARRANTY, to the extent permitted by law.

   Written by Mike Haertel and others; see
   <https://git.sv.gnu.org/cgit/grep.git/tree/AUTHORS>.
   ----- jq version -----jq-1.6
   ----- sed version -----
   sed (GNU sed) 4.8
   Packaged by Debian
   Copyright (C) 2020 Free Software Foundation, Inc.
   License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
   This is free software: you are free to change and redistribute it.
   There is NO WARRANTY, to the extent permitted by law.

   Written by Jay Fenlason, Tom Lord, Ken Pizzini,
   Paolo Bonzini, Jim Meyering, and Assaf Gordon.

   This sed program was built with SELinux support.
   SELinux is disabled on this system.

   GNU sed home page: <https://www.gnu.org/software/sed/>.
   General help using GNU software: <https://www.gnu.org/gethelp/>.
   E-mail bug reports to: <bug-sed@gnu.org>.
   ----- curl version -----
   curl 7.81.0 (x86_64-pc-linux-gnu) libcurl/7.81.0 OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.13
   Release-Date: 2022-01-05
   Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
   Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets zstd

   ----- is this a sha instead of a tag? false  -----
   ----- pulling image registry-1.docker.io/gravitl/netmaker:v0.16.0 -----
   ----- digest: sha256:5bd4cb183884b66c8f2c9d33c2111f1f22bb520466af5e28c2262256e194dc92 -----
   ----- new digest: sha256:12e79f3d35400281ad1ac4e695f2a4b4dc04e546208e687be3c0d289666671de -----
   ----- final manifest: {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "schemaVersion": 2,
      "config": {
         "mediaType": "application/vnd.docker.container.image.v1+json",
         "digest": "sha256:26456d1071304fc69d3a2befeaa226965ec02181db3b41f6dbcbac59e124f65e",
         "size": 2470
      },
      "layers": [
         {
            "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "digest": "sha256:3aa4d0bbde192bfaba75f2d124d8cf2e6de452ae03e55d54105e46b06eb8127e",
            "size": 2812689
         },
         {
            "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "digest": "sha256:fa8c39cc66f7357c866113064370618b235e2335f3a879aff3814a19d4e27da7",
            "size": 4926729
         },
         {
            "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1",
            "size": 32
         },
         {
            "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "digest": "sha256:c437bf191b517f33398e9aa17a9a7f869f0cb3fd9476ba522dae8fe199fa7a07",
            "size": 144
         },
         {
            "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "digest": "sha256:5fd79929c39312b20b3bd620dd027489782ccf9b2dff60382dba77400365dc8c",
            "size": 5497107
         },
         {
            "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "digest": "sha256:af9ad32dc18ffb0256cdd1fc08f9a4c7a9679d0a8a3212e1eeee6ae06d162e15",
            "size": 2107
         }
      ]
   } -----
   ----- blob_digest: sha256:3aa4d0bbde192bfaba75f2d124d8cf2e6de452ae03e55d54105e46b06eb8127e -----
   sed: -e expression #1, char 73: Invalid range end
   ----- blob_digest: sha256:fa8c39cc66f7357c866113064370618b235e2335f3a879aff3814a19d4e27da7 -----
   sed: -e expression #1, char 73: Invalid range end
   ----- blob_digest: sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 -----
   sed: -e expression #1, char 73: Invalid range end
   ----- blob_digest: sha256:c437bf191b517f33398e9aa17a9a7f869f0cb3fd9476ba522dae8fe199fa7a07 -----
   sed: -e expression #1, char 73: Invalid range end
   ----- blob_digest: sha256:5fd79929c39312b20b3bd620dd027489782ccf9b2dff60382dba77400365dc8c -----
   sed: -e expression #1, char 73: Invalid range end
   ----- blob_digest: sha256:af9ad32dc18ffb0256cdd1fc08f9a4c7a9679d0a8a3212e1eeee6ae06d162e15 -----
   sed: -e expression #1, char 73: Invalid range end
ingshtrom commented 1 year ago

I am sorry for the delay here. We aren't quite sure what is going on here. You have found a workaround with the VPN for now, at least 😓

We have an NLB doing TLS termination in our architecture and as part of some parallel work, we are turning on access logs for it. I will let you know when we have that enabled and ping you to send some more requests without the VPN in order to help diagnose what is happening.

I am sorry for this and I appreciate your patience.

ingshtrom commented 1 year ago

We have the access logs enabled. If you could try pulling images again several times without the VPN that would be helpful in us debugging this further.

Thank you!

projx commented 1 year ago

Hi

I've just done this, trying to pull jc21/nginx-proxy-manager:latest

ERROR: Get "https://registry-1.docker.io/v2/": read tcp 10.10.40.44:57404->3.216.34.172:443: read: connection reset by peer
root@dcker-01:~/npm# docker-compose up
Pulling app (jc21/nginx-proxy-manager:latest)...
ERROR: Head "https://registry-1.docker.io/v2/jc21/nginx-proxy-manager/manifests/latest": Get "https://auth.docker.io/token?scope=repository%3Ajc21%2Fnginx-proxy-manager%3Apull&service=registry.docker.io": read tcp 10.10.40.44:54418->34.194.164.123:443: read: connection reset by peer
root@dcker-01:~/npm# docker-compose up
Pulling app (jc21/nginx-proxy-manager:latest)...
ERROR: Head "https://registry-1.docker.io/v2/jc21/nginx-proxy-manager/manifests/latest": read tcp 10.10.40.44:57414->3.216.34.172:443: read: connection reset by peer
root@dcker-01:~/npm# docker-compose up
Pulling app (jc21/nginx-proxy-manager:latest)...
ERROR: Get "https://registry-1.docker.io/v2/": read tcp 10.10.40.44:58302->3.216.34.172:443: read: connection reset by peer
root@dcker-01:~/npm# docker-compose up
Pulling app (jc21/nginx-proxy-manager:latest)...
ERROR: Get "https://registry-1.docker.io/v2/": read tcp 10.10.40.44:58318->3.216.34.172:443: read: connection reset by peer
root@dcker-01:~/npm# docker-compose up
Pulling app (jc21/nginx-proxy-manager:latest)...
ERROR: Head "https://registry-1.docker.io/v2/jc21/nginx-proxy-manager/manifests/latest": Get "https://auth.docker.io/token?scope=repository%3Ajc21%2Fnginx-proxy-manager%3Apull&service=registry.docker.io": read tcp 10.10.40.44:58190->34.194.164.123:443: read: connection reset by peer
root@dcker-01:~/npm# docker-compose up
Pulling app (jc21/nginx-proxy-manager:latest)...
ERROR: Get "https://registry-1.docker.io/v2/": read tcp 10.10.40.44:58330->3.216.34.172:443: read: connection reset by peer

Then I apply the VPN routing, and it pulls fine:

root@dcker-01:~/npm# docker-compose up
Pulling app (jc21/nginx-proxy-manager:latest)...
latest: Pulling from jc21/nginx-proxy-manager
f003217c5aae: Pull complete
ce6a15c1ccfb: Pull complete
b7091728beb7: Pull complete
69285662a50c: Pull complete
8ffff58f8338: Pull complete
73a8411ba8dd: Pull complete
08a1d8a12d96: Pull complete
3ad728d3ce37: Pull complete
d9e02947749c: Pull complete
ingshtrom commented 1 year ago

I tested the IP you are getting during the docker pull, 3.216.34.172, and that is not one of the current public IPs being returned. Here is what I get from dig

dig registry-1.docker.io +short
18.215.138.58
34.194.164.123
52.1.184.176

NOTE: These public IPs do rotate, but it shouldn't be too often. We don't keep track, but that 3.216.34.172 has been in this thread from the beginning of your problems.

Can you try modifying your /etc/hosts with one of the IPs I provided above?

The other thing that came to mind is trying another domain. The Docker Hub registry is actually available behind multiple domains, so you can try pulling the image registry.docker.com/alpine:latest or index.docker.io/alpine:latest. All of these domains, registry-1.docker.io, index.docker.io, and registry.docker.com should resolve to the same public IP addresses, though. I don't expect a difference, but I am curious if it helps.

[edit]: We don't see these requests in our NLB logs--only the requests of you going to hub.docker.com 🤔 I'm triple checking this data to make sure, sorry for responding too quickly on this part.

ingshtrom commented 1 year ago

It is weird. We don't see requests for anything related to nginx or nginx-proxy-manager in the last couple of days. We see some downloads of images in the traefik and rancher namespaces. In addition, I see a bunch of seemingly random tools you might be using for miscellaneous tasks. Based on the data, these requests for rancher and traefik images seem to be downloaded without the VPN?

I'm glad the VPN workaround is working. Since we aren't seeing the requests for the nginx test download you did, there isn't much else we can do, unfortunately.

projx commented 1 year ago

Thanks for checking - I've contacted my ISP and asked them to change my IP.. things are working!