GHCR rate limits - Githubissues

Description

Release workflows fail somewhat regularly with rate-limiting errors:

Error: signing [ghcr.io/chainguard-images/git@sha256:9b4afea457fb39e297f83ec71e35f4094badbe8d49bab6ce9af5ae55734050a2
ghcr.io/chainguard-images/git@sha256:4e7eae52cad8463c7d19c40fc821a144b06472fddb431431f0da430a0d123270
ghcr.io/chainguard-images/git@sha256:8be1d6ce98579d65b04b57b749e7cef7dd261a6a51854f5d4aae2f5ce241236a]: signing digest:
POST https://ghcr.io/v2/chainguard-images/git/blobs/uploads/: TOOMANYREQUESTS: retry-after: 10.305464ms, allowed: 2000/minute

(link)

The broader workflow run that hit this is here: https://github.com/chainguard-images/images/actions/runs/3830355184 -- it's got 55 workflows running in parallel, most of which seem to complete within 5-10 minutes, when they don't hit the limit.

Notes:

this is coming from GHCR, not Dockerhub -- though I think we've seen DH pull rate limits too
this is happening on pushes, not pull
the response includes data about when we should be able to retry, if we think we might be able to extract it to retry
the retry-after above is actually not that much -- if we just waited 10ms, we wouldn't have hit the ceiling?
GitHub's rate limit API doesn't seem to include package pull/push rate limit info

Since we're now building all our images in separate parallel workflow runs, we might be able to just stagger them all by a random number of seconds up to a minute or two, just so we don't stampede in the first few minutes when the workflow is kicked off.

chainguard-images / images

GHCR rate limits #173