Docker pull limits coming Nov 2, 2020

[Codelica]

Only posting this as a potential bug for review since a change is coming to Dockerhub on Nov 2, 2020.

Describe the bug Docker is instituting pull rate limits for anonymous and free users starting Nov 2, 2020. Per their email:

You are receiving this email because of a policy change to Docker products and services you use. On Monday, November 2, 2020 at 9am Pacific Standard Time, Docker will begin enforcing rate limits on container pulls for Anonymous and Free users. Anonymous (unauthenticated) users will be limited to 100 container image pulls every six hours, and Free (authenticated) users will be limited to 200 container image pulls every six hours, when enforcement is fully implemented. Docker Pro and Team subscribers can pull container images from Docker Hub without restriction, as long as the quantities are not excessive or abusive.

Considering Watchtower has the following warning posted about it's --monitor-only flag:

Please note Due to Docker API limitations the latest image will still be pulled from the registry.

I'm guessing a pull is done to see if there is a newer version?

Anyway, I guess the question is if Docker considers a pull "attempt" (where there isn't a newer image) any different than a pull of a newer image. I haven't found any info on that yet.

[github-actions[bot]]

Hi there! 👋🏼 As you're new to this repo, we'd like to suggest that you read our code of conduct as well as our contribution guidelines. Thanks a bunch for opening your first issue! 🙏

[simskij]

Hey @Codelica,

Thanks for the heads up. I'm trying to get in contact with the dockerhub team to explore our options here. From the docker blog I get the impression that even a manifest fetch will count towards the rate limit, and in that case our options are kind of limited. Hopefully, we'll find a reasonable way around this however, ,given the popularity of this project.

Best, Simme

[TheCatLady]

I think the solution is to just adjust the default poll interval to something much longer than the current default of 5 minutes, and maybe some documentation on what counts as a docker pull so that users can adjust accordingly based on their usage.

[albertoxamin]

Perhaps the auth should be put on the getting started page of the docs

[simskij]

I've gone through the docker client code, and wrote about my findings in https://github.com/containrrr/watchtower/discussions/668#discussioncomment-116021

The TL;DR is that it already handles this and that watchtower checking for image updates shouldn't count towards the rate limit, as only the SHA digest is fetched using an HTTP request type that's excluded from the rate limit.

[bolmstedt]

I'm just started seeing a lot of rate limit errors 15 minutes ago, so I think this is an actual issue.

Unable to update container "/opt_watchtower_1": Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. Proceeding to next.

[albertoxamin]

Same here, my log is filled with rate limit errors

[tombomb]

It looks like something may have changed as I was getting lots of end of file errors till this morning (something I have been too busy to troubleshoot) to now getting the rate limits when trying to check for updates

[chosten]

Came here to say the exact same thing, logs are filling fast, I have to disable Watchtower.

[eikendev]

As an intermediate fix, we can manually set the interval argument. Docker will eventually go down to a limit of 100 requests per 6 hours for anonymous users. If you run c containers, Watchtower will do c / i checks per second (on average), where i is the interval you pass. Thus, we need to make sure that 100 / 6 >= 60 * 60 * c / i, or equivalently that i >= 6 * 60 * 60 * c / 100.

In short, given your number of containers c, you need to pass the interval i = 216 * c, or better something a bit above that.

[piksel]

@eikendev our investigations into this concluded that this shouldn't be the case unless there were actual changes. Unfortunately, this was only what we gathered from reading the source and talking to the Docker developers. We haven't actually been able to test this (and it seems to happen to users randomly).

If docker actually does a manifest pull every check, then we need to find another solution...

But yeah, set the interval higher if you encounter the limit, and I guess we need to do a wireshark capture to really get to the bottom of what is going on...

[TheCatLady]

Just FYI, according to https://www.docker.com/increase-rate-limits, we are currently in the middle of the "temporary full enforcement window," which is why the errors are surfacing now:

Current Docker Hub Usage Limit Status (updated 11/2,4:00pm Pacific) Unauthenticated requests: | 5,000 per six hours Free tier requests: | 5,000 per six hours Temporary full enforcement window (100 per six hours for unauthenticated requests, 200 per six hours for free accounts): November 4, 9am-12 noon Pacific Time.

[simskij]

To all of you who are reaching the rate limit: Can you please let us know the version of your docker installation? I've been running watchtower with a 1s interval now for a good ten minutes without hitting the limit. Just reached it 😩

[aveao]

Docker version 19.03.13, build 4484c46d9d

Edit: Just saw your edit.

[simskij]

Ok, so this is apparently not a watchtower issue (not only at least). Instructing the docker engine to check for new images incurs the rate limit when using the unix socket directly as well:

❯ curl --unix-socket /var/run/docker.sock http://docker/images/create\?fromImage\=containrrr/watchtower\&tag\=latest -X POST                                                                                                                                                                                                                                                                                                                                                                                                                                                140ms
{"message":"toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit"}

I have the latest image downloaded already. Will continue to look into this. Meanwhile, please extend your intervals to avoid running into the limit.

[PrivatePuffin]

Just hit this issue as well, I didn't download any docker containers myself during the full enforcement limit, besides watchtower and seemingly still hit the limit (with <40 containers running)

[simskij]

The hard rate limit @TheCatLady was mentioning has now been lifted again. The rate limit for anonymous usage right now is 2500 per 6 hours, so you should still adjust your interval accordingly to limit the risk of being rate limited.

[PrivatePuffin]

The hard rate limit @TheCatLady was mentioning has now been lifted again. The rate limit for anonymous usage right now is 2500 per 6 hours, so you should still adjust your interval accordingly to limit the risk of being rate limited.

Just the fact the temporary enforcement window is gone now, doesn't mean you where right. You where clearly wrong in your assumption that Watchtower checks wouldn't count afaik.

[TheCatLady]

The hard rate limit @TheCatLady was mentioning has now been lifted again. The rate limit for anonymous usage right now is 2500 per 6 hours, so you should still adjust your interval accordingly to limit the risk of being rate limited.

Just the fact the temporary enforcement window is gone now, doesn't mean you where right. You where clearly wrong in your assumption that Watchtower checks wouldn't count afaik.

Err, @simskij literally commented that you should adjust the poll interval to avoid getting rate-limited...

[PrivatePuffin]

Err, @simskij literally commented that you should adjust the poll interval to avoid getting rate-limited...

He also not long ago pointed towards #668 , which was actually why I hit the rate limit in the first place. (which wasn't too clearly rescinded as far as I can tell.)

So, if I understand correctly after reading this thread twice now, what he meant previously was the documented behavior of docker (on wich we previously based the assumption that watchtower wouldn't hit the limit) doesn't seem to be what is actually currently hapening?

---

edit Basically: yes I understand correctly it seems. Resolved (see https://github.com/containrrr/watchtower/issues/669#issuecomment-721969780 )

[simskij]

The hard rate limit @TheCatLady was mentioning has now been lifted again. The rate limit for anonymous usage right now is 2500 per 6 hours, so you should still adjust your interval accordingly to limit the risk of being rate limited.

Just the fact the temporary enforcement window is gone now, doesn't mean you where right.

You where clearly wrong in your assumption that Watchtower checks wouldn't count afaik.

Thanks for the constructive feedback! To my knowledge, this is still an open-source project where we do as good as we can with the limited spare time we have. Pointing fingers is not helping.

If you have the need for better SLAs, I would suggest you to find another solution.

You get the service for free, which also means we're not leaving any guarantees what so ever in terms of stability.

The assumption, as you so elegantly describe it, was based on following the code in the docker engine repo as well as discussing the issue with docker's Head of Open Source.

Given we couldn't actually do any proper testing beforehand, my view has not changed; we did the best we could with the information we had.

We will of course try to solve this as soon as possible, but until then, I'll just repeat:

Adjust your poll interval or you will likely hit the rate limit.

[piksel]

@Ornias1993 Basically, yes. There is a way to check without getting the whole manifest and is exempted from the rate limiting. The Docker source shows that it should check that before doing the actual manifest pull. But now that the rate limit is enforced, it seems like docker doesn't just do that. We still don't fully understand why that is.

[PrivatePuffin]

Maybe I was a bit harsh, I'm not expecting an SLA or guaranteed stability obviously. I just was a bit struck by the lack of saying "sorry" when someone makes a mistake. Could be a cultural thing though.

Anyhow, @piksel Thats very very odd indeed... So the question really is "what is rate limited" atm...

[simskij]

This PR in moby/moby aims to resolve the issue as there is an issue with how the docker engine caches manifest-lists.

https://github.com/moby/moby/pull/41607

However, we might need to investigate alternative solutions for this in the meantime as it will take a while before this makes it to a patch release and then finds its way out to all docker installations running watchtower.

[piksel]

Yeah, I guess. From the API's point of view, it's clear that it's the HEAD requests that are "free", and using such a request is enough to check if a tag has changed. If docker doesn't fix this upstream we might have to go another route.

Regarding the "sorry": anyone has been free to investigate how this would pan out. This is a community project and we try to make software that helps us by collaborating. We are of course sorry for the inconvenience this causes us users, but not for "getting it wrong".

[simskij]

We are of course sorry for the inconvenience this causes us users, but not for "getting it wrong".

Sums it up beautifully. Well expressed, @piksel!

@Ornias1993:

There definitely is a bit of stress involved when this kind of project goes south given the magnitude of it's usage, as you probably can imagine. 😅

Hours of following the code back and forth, trying to understand why the docker engine is not behaving as it should, knowing everyones log files continued to grow, clearly took its toll on my temper.

I apologize for interpreting your post as hostile if that wasn't your intention.

[PrivatePuffin]

I apologize for interpreting your post as hostile if that wasn't your intention.

No hard/harsh feelings from my side... :) Lets get this back on track!

[deividaspetraitis]

I was wondering, is it possible currently to adjust/set poll interval to specific images ( haven't found such option in docs ). For example do use both: public and private registries. So I would like to pull every 30secs for my private, but in public hub case set to 5mins to not hit the limit. In case there is no such feature at the moment could this idea receive support from the community for implementation?

[simskij]

I was wondering, is it possible currently to adjust/set poll interval to specific images ( haven't found such option in docs ). For example do use both: public and private registries. So I would like to pull every 30secs for my private, but in public hub case set to 5mins to not hit the limit. In case there is no such feature at the moment could this idea receive support from the community for implementation?

I know we have a feature request about it somewhere, but currently we lack the support I'm afraid. Sorry. 😩

[cpuguy83]

https://github.com/tianon/dockerhub-public-proxy/blob/master/dockerhub-public-proxy.pl

This can be used to cache manifests... even to something like a free cloudfare. One of the things it does is redirect the GET request for a manifest to a HEAD, then a GET to the cacheable/immutable, content addressed manifest url.

There are other solutions as well, but ultimately converting to a HEAD when using mutable tags is needed.

https://github.com/rpardini/docker-registry-proxy is another project that just uses nginx to cache all the things and sets cache expiration for mutable URLs. Has a fallback to use stale caches when hub returns a 429 (or is just down).

[lukjod]

How to put credentials for docker to have bigger limit on every watchtower run?

[PrivatePuffin]

@lukjod Just login to docker cli: docker login -u USERNAME

[lukapeschke]

@Ornias1993 I think you meant @lukjod :smile:

[PrivatePuffin]

SHOOT. I hate the fact github sometimes completely removes certain names from the dropdown.... >.<

[lukjod]

thx should i do it just once or with creation of watchtower docker image. Sorry not too familiar with docker

[PrivatePuffin]

Watchtower uses the host docker instance, so just once on the host ;)

[victorcmoura]

@deividaspetraitis

I was wondering, is it possible currently to adjust/set poll interval to specific images ( haven't found such option in docs ). For example do use both: public and private registries. So I would like to pull every 30secs for my private, but in public hub case set to 5mins to not hit the limit. In case there is no such feature at the moment could this idea receive support from the community for implementation?

As a temporary workaround, you can run multiple Watchtower instances and define a monitoring scope for each one (basically setting which containers will belong to which scope). With this setup, you will be able to configure different intervals for every scope.

[simskij]

Stuff is happening in https://github.com/containrrr/watchtower/pull/674

[ilike2burnthing]

Rate limit now down to 1,000 per six hours - https://www.docker.com/increase-rate-limits

Also:

Temporary full enforcement windows (100 per six hours for unauthenticated requests,
200 per six hours for free accounts): November 9, 9am-3pm Pacific time and
November 10, 3am-9am Pacific time.

[omar-dulaimi]

Hello all, I'm not sure I get how the newly introduced rate limits could affect an aws pipeline that uses ecr and docker(not docker hub). Could somebody explain why this is affecting us?

[deividaspetraitis]

@omar-dulaimi

Hello all, I'm not sure I get how the newly introduced rate limits could affect an aws pipeline that uses ecr and docker(not docker hub). Could somebody explain why this is affecting us?

watchtower`s image itself is published on public hub, I guess that's why.

[omar-dulaimi]

@deividaspetraitis So this means watchtower is being used internally with pull and push commands since we don't explicitly mention watchtower anywhere in our scripts. Thanks.

[piksel]

@omar-dulaimi I think you need to explain the issue you're experiencing, (and perhaps in another ticket, since, as you say, this doesn't seem to be the same issue).

[joshghent]

Weirdly I've ran docker login and got the following config in /home/josh/.docker/config.json

"auths": {
    "https://index.docker.io/v1/": {
        "auth": "xxx"
    }
}

I've then passed this in the docker-compose.yml file

services:
  watchtower_apps:
    command: watchtower archivewarrior
    container_name: watchtower_apps
    environment:
      WATCHTOWER_CLEANUP: "true"
      WATCHTOWER_DEBUG: "true"
      WATCHTOWER_ROLLING_RESTART: "true"
    image: v2tec/watchtower
    restart: unless-stopped
    volumes:
    - /home/josh/.docker/config.json:/config.json:rw
    - /var/run/docker.sock:/var/run/docker.sock:rw
version: '3.8'

But am still getting the error Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading. This seems to indicate that the request is not being made authenticated? Anyone have any ideas?

[piksel]

@joshghent Could you open a separate discussion for this? I know it's related, but most people track this for updates on the main issue.

[TheCatLady]

@joshghent The message text is misleading, but I believe Docker Hub returns the same error response whether or not you're authenticated.

[joshghent]

@joshghent Could you open a separate discussion for this? I know it's related, but most people track this for updates on the main issue.

Will do, apologies for polluting the discussion. Thank you for a great project 👍

[nicx]

any news when this will be solved? anything we can do?

[ilike2burnthing]

@nicx https://github.com/containrrr/watchtower/issues/669#issuecomment-721885754

Currently the limit is 1000 per 6 hours, so set your interval to greater than or equal to 6 60 60 * c / 1000.

As for a fix, keep an eye on - https://github.com/containrrr/watchtower/pull/674

[nicx]

thanks @ilike2burnthing for clarification. Just to be correct, I think you have a typo in your formula: It should be "100" instead of "1000", so the correct formula is 6 * 60 * 60 * c / 100