Docker Hub rate limiting is making CI tests flaky

[moreati]

e.g. https://travis-ci.org/github/dw/mitogen/jobs/753779697

ff50d722b382: Retrying in 2 seconds ff50d722b382: Retrying in 1 second toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit The command ".ci/${MODE}_install.py" failed and exited with 1 during .

From https://www.docker.com/increase-rate-limit

On November 20, 2020, rate limits anonymous and free authenticated use of Docker Hub went into effect. Anonymous and Free Docker Hub users are limited to 100 and 200 container image pull requests per six hours.

Possible mitigations or fixes

• Use authenticated requests. Currently Mitogen requests are counted along with all other anonymous requests from the same IP.
  • Who's account?
• Upgrade to a premium Docker account
  • Paid for by who?
  • Can we embed a secret/token in the repo, given constraints?
• Switch to an alternative container registry
• Reduce Docker Registry usage
  • Perhaps with caching, or pinning?

[moreati]

Attempts so far to mitigate

• 789 failed because PRs (rightly) do not have access to Travis or Azure Pipeline secrets. This is a sensible policy, but unfortunate.

• 790 Successful, but of limited benefit

[moreati]

Idea number 3 is a mirror of https://hub.docker.com/u/mitogen, hosted on AWS Elastic Container Registry. The base URL is public.ecr.aws/n5z0e8q9, and available images can be browsed at https://gallery.ecr.aws/?searchTerm=n5z0e8q9. The mirror is currently a one time snapshot, updates on Docker Hub will not automatically appear on AWS ECR. I believe the free tier will be sufficient for our needs

As a new or existing customer, Amazon ECR offers you 50 GB-month of always-free storage for your public repositories. You can transfer 500 GB of data to the internet for free from a public repository each month anonymously -- https://aws.amazon.com/ecr/pricing/

The images were synchronised using skopeo

``` :~ $ aws ecr-public get-login-password --region us-east-1 | skopeo login --username AWS --password-stdin public.ecr.aws/n5z0e8q9 Login Succeeded! :~ $ skopeo sync --insecure-policy --src docker --dest docker docker.io/mitogen/centos5-test public.ecr.aws/n5z0e8q9 INFO[0000] Tag presence check imagename=docker.io/mitogen/centos5-test tagged=false INFO[0000] Getting tags image=docker.io/mitogen/centos5-test INFO[0001] Copying image tag 1/1 from="docker://mitogen/centos5-test:latest" to="docker://public.ecr.aws/n5z0e8q9/centos5-test:latest" Getting image source signatures Copying blob cfa7a04d855c done Copying blob 3e745e61e851 done Copying blob 2068b24f564b done Copying config bb41b50af1 done Writing manifest to image destination Storing signatures INFO[0099] Synced 1 images from 1 sources :~ $ skopeo sync --insecure-policy --src docker --dest docker docker.io/mitogen/centos6-test public.ecr.aws/n5z0e8q9 INFO[0000] Tag presence check imagename=docker.io/mitogen/centos6-test tagged=false INFO[0000] Getting tags image=docker.io/mitogen/centos6-test INFO[0001] Copying image tag 1/1 from="docker://mitogen/centos6-test:latest" to="docker://public.ecr.aws/n5z0e8q9/centos6-test:latest" Getting image source signatures Copying blob 949c1d7640c4 done Copying blob ff50d722b382 done Copying config a941b442e1 done Writing manifest to image destination Storing signatures INFO[0073] Synced 1 images from 1 sources :~ $ skopeo sync --insecure-policy --src docker --dest docker docker.io/mitogen/centos7-test public.ecr.aws/n5z0e8q9 INFO[0000] Tag presence check imagename=docker.io/mitogen/centos7-test tagged=false INFO[0000] Getting tags image=docker.io/mitogen/centos7-test INFO[0001] Copying image tag 1/1 from="docker://mitogen/centos7-test:latest" to="docker://public.ecr.aws/n5z0e8q9/centos7-test:latest" Getting image source signatures Copying blob 7c1993ee599d done Copying blob 8ba884070f61 done Copying config e36619bcd2 done Writing manifest to image destination Storing signatures INFO[0069] Synced 1 images from 1 sources :~ $ skopeo sync --insecure-policy --src docker --dest docker docker.io/mitogen/debian-test public.ecr.aws/n5z0e8q9 INFO[0000] Tag presence check imagename=docker.io/mitogen/debian-test tagged=false INFO[0000] Getting tags image=docker.io/mitogen/debian-test INFO[0001] Copying image tag 1/1 from="docker://mitogen/debian-test:latest" to="docker://public.ecr.aws/n5z0e8q9/debian-test:latest" Getting image source signatures Copying blob 54f7e8ac135a done Copying blob d03011a89050 done Copying config 3d20cb66f4 done Writing manifest to image destination Storing signatures INFO[0086] Synced 1 images from 1 sources ```