search

docker / hub-feedback

Feedback and bug reports for the Docker Hub
https://hub.docker.com
233 stars 48 forks source link

Entire Linode /32 seems to share a single rate limit for unauthenticated image pulls #2339

Closed adamenhance closed 10 months ago

adamenhance commented 10 months ago

It seems that this behaviour began in the past 24 hours. 

root@localhost:~# TOKEN=$(curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" | jq -r .token)
root@localhost:~# TOKEN=$(curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" | jq -r .token)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  4415    0  4415    0     0  12598      0 --:--:-- --:--:-- --:--:-- 12614
root@localhost:~# curl --head -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest
HTTP/1.1 200 OK
content-length: 2782
content-type: application/vnd.docker.distribution.manifest.v1+prettyjws
docker-content-digest: sha256:767a3815c34823b355bed31760d5fa3daca0aec2ce15b217c9cd83229e0e2020
docker-distribution-api-version: registry/2.0
etag: "sha256:767a3815c34823b355bed31760d5fa3daca0aec2ce15b217c9cd83229e0e2020"
date: Wed, 23 Aug 2023 17:04:23 GMT
strict-transport-security: max-age=31536000
ratelimit-limit: 100;w=21600
ratelimit-remaining: 0;w=21600
docker-ratelimit-source: 2a01:7e00::

As far as I can tell this means that any Linode VPS in this /32 pulling images over ipv6 will always be denied.

root@localhost:~# docker pull ubuntu
Using default tag: latest
Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
ingshtrom commented 10 months ago

I am sorry you are having trouble! Let me see if I can clear things up.

We rolled out IPv6 today and in the blog post we describe how we are rate limiting IPv6 IP addresses against the first 64 bits in the address, so if you have an IP address of 1111:2222:3333:4444:5555:6666:7777:8888, then we would rate limit against 1111:2222:3333:4444:xxxx:xxxx:xxxx:xxxx. The second half of the IP address is masked.

In your case, the docker-ratelimit-source header says 2a01:7e00::, which is actually a valid, compressed format which represents 2a01:7e00:0000:0000:xxxx:xxxx:xxxx:xxxx.

In this situation, the following IP addresses would be rate limited into the same bucket:

The following IP addresses would be rate limited into separate buckets:

Therefore, I think the rate limiting is being done as our blog is saying, but you are seeing the compressed format of your masked address in the header, rather than it fully being written out with all zeros.

adamenhance commented 10 months ago

Thanks for clarifying, I wasn't actually aware that ipv6 support was new.

So the maximum prefix length is /64? The problem here is that Linode by default use SLAAC, to get a dedicated /64 you have to deliberately enable it. We only use Linode for CI so this can be easily worked around but it might affect other users of Linode and similar providers who are unaware of this policy.

tchinmai7 commented 10 months ago

Linode is actively investigating this - https://status.linode.com/. Please reach out to support@linode.com with contact information so we can follow up directly

ingshtrom commented 10 months ago

This should be resolved now. We are working closely with Linode/Akamai in order to make sure we are aligned on how they use Docker Hub to avoid issues like this in the future.