search

lf-edge / eve

EVE is Edge Virtualization Engine
https://www.lfedge.org/projects/eve/
Apache License 2.0
468 stars 158 forks source link

CI/CD: hitting DockerHub pull rate limit a.k.a. TOOMANYREQUESTS #4008

Open uncleDecart opened 1 week ago

uncleDecart commented 1 week ago

This is known CI/CD issue affecting our building and testing pipeline both on Eden and EVE side. DockerHub has pull rate limits as of November 2020, BuildJet runners, which are currently used in Eden and EVE. In their documentation, BuildJet recommends here to authenticate with Docker account to mitigate pull rate limits, since anonymous DockerHub pulls are shared across all the actions run on BuildJet. That would give us 200 pulls daily, compared to 100 daily pulls per anonymous address shared with others. We should authenticate, it'll help us, but this solution is still not sustainable. Note that this pull rate does not apply to some containers, such as LF-Edge containers. Currently known affected images are Redis, registry, alpine, nginx. Packing and distributing them under lf-edge would solve the problem, but I'm not sure we can do that because of licensing.

Previously migrating to ghcr was a viable option, since it didn't have pull rate limit According to this discussion ghcr now also has pull rate limit.

I'm opening this issue to keep track on this problem.

Related PRs:

giggsoff commented 1 week ago

As I remember previously we solved the problem having sposored oss bage. Is it still help us to not hit the limit at least for the part of repos?

uncleDecart commented 1 week ago

As I remember previously we solved the problem having sposored oss bage. Is it still help us to not hit the limit at least for the part of repos?

@giggsoff redis, registry, alpine, nginx can't have sponsored oss badge. In theory, we can repackage them under lf-edge, but I think we might have a problem with license

OhmSpectator commented 1 week ago

Can we pull the images from some local registry, at least for tests?.. =(

deitch commented 1 week ago

Distributing them under OSS will create all sorts of licensing issues, as well as potential issues for lf-edge itself.

What about storing them somehow on our own? Could we package them up as tar files and then load them into local docker for tests? So we pull them exactly once, save them as tar, and then use them in the tests? I am not quite sure where we would store them. GHA artifacts don't quite work, as they have limits. Maybe ghcr like you suggested, or Docker Hub, but limited to pulling just for these actions? The problem is that these are all OSS, so they would need to be open to anyone, wouldn't they, recreating the licensing and distribution issues?

uncleDecart commented 1 week ago

What about storing them somehow on our own?

Where would you store them? EVE is an open source product, so storage should not be affiliated with any commercial company AFAIU

Maybe ghcr like you suggested, or Docker Hub, but limited to pulling just for these actions

Yes, ghcr now have pull rate limit. I think GitHub artefacts don't have pull rate limit, so packaging and sharing them would help us. But if I get it right, docker pricing policy is 5 dollars per month for 5000 pulls per day

The problem is that these are all OSS, so they would need to be open to anyone, wouldn't they, recreating the licensing and distribution issues?

Exactly