Dockerhub rate limits issue when using Private EC2s

[sanderegg]

Since AWS-master and Tip.science are using private EC2 machines the following issue is arising:

• every machine in these respective deployment are now pulling docker images from Dockerhub via the same IP address instead of multiple (when they were public),
• the Dockerhub rate limit blocks the pulling of images (100 pulls every 6 hours),
• this blocks autoscaled machines and external clusters after some usage,

Current possible options:

• ensure docker login to Dockerhub is done on the machines as a custom boot script (works but clusters-keeper UserData is already almost at 16KB, which is the max limit),
• change how Clusters-keeper handle these scripts by leveraging AWS SSM and send command over them (this aleviates the 16KB issue)
• set up the internal registry to act a pull through cache to Dockerhub, then pulling images via our registry would pull from Dockerhub only once and otherwise keep the images in cache, therefore reducing the trafic between our machines and Dockerhub, which should also lower our costs.

some references:

• https://docs.aws.amazon.com/AmazonECR/latest/userguide/pull-through-cache.html
• https://docs.docker.com/docker-hub/mirror/

### Tasks
- [ ] master configuration
- [ ] staging configuration
- [ ] prod configuration

[YuryHrytsuk]

Literature

• Docker Hub rate limit
  • Rate limit numbers
  • How to authenticate pull requests in different contexts
  • How to check my current (+remaining) limits

[YuryHrytsuk]

Docker log in per request vs Private registry as pull through cache to Dockerhub

Docker log in per request - need to be done per request (every time) - need to log in on every machine

Private registry as pull through cache to Dockerhub + once set up works automatically out-of-the-box with no overhead - OPS need to guarantee private registry uptime - OPS need to properly loadbalance and scale private registry - OPS need to maintain disk space of private registries - once registry is down no image can be pulled via regsitry - daemon.json needs to be configured on every machine

Decision Based on this comparison, I decide to go on with Docker log in per request approach. Even though I would rather have something working automatically out-of-the-box but the cost is too high

@mrnicegyu11, @sanderegg, let me know what are your thoughts on this matter

[YuryHrytsuk]

Docker swarm and authenticated image pulling. Questions

Does it matter which user "runs" docker container [linux] process? (unrelated to docker swarm)

• It does not. As long as you issue docker command with authenticated user and --with-registry-auth option, the pulling of images is authenticated

Do I need to issue docker login on every docker swarm node?

• According to this SO Answer, you do not need to be logged in on every machine. You need to issue docker login on the machine from where you run docker command (e.g. docker service create --with-registry-auth ...

Do I need to issue docker login every time I ssh into machine?

• Based on experiments conducted below, I can say NO. Docker will rely on ~/.docker/config.json file apparently

Do I need to issue docker login on the machine or docker swarm already have login information (in case it was run before with --with-registry-auth)?

• Docker swarm already hove login information, if it was set up properly (docker login + --with-registry-auth option)

Solution. The way to go

1. Make sure docker is authenticated for the user that runs the commands mentioned in the step below
2. Execute all docker service ... and docker stack ... commands with --with-registry-auth option
3. Have a test monitoring remaining pull requests for anonymous user and report in case it gets below 100 (the max number)

Docker Swarm Image Pulling experiments

Swarm Cluster:

• 1 manager
• 1 ops
• 1 simcore docker

Case 1

Setup:

• docker is logged in on manager only.
• command: docker service create --mode global --name hello-world hello-world is executed on manager

Outcome:

• 4 (I expected 3 ...) docker pull requests were taken from anonymous user
  • it could be due to pulling in autoscaled machines (the testing cluster reused infrastructure of AWS Master)

Case 2

Setup:

• docker is logged in on manager only.
• command: docker service create --mode global --name hello-world hello-world is executed on manager

Outcome:

• 2 (I expected 3 ...) docker pull requests were taken from anonymous user

Case 3

Setup:

• docker is logged in on manager only.
• command: docker service create --mode global --name hello-world hello-world is executed on manager

Outcome:

• 2 docker pull requests were taken from anonymous user

Case 4

Setup:

• docker is logged in on manager only.
• command: docker service create --with-registry-auth --mode global --name hello-world hello-world is executed on manager

Outcome:

• 0 docker pull requests were taken from anonymous user

Case 4

Setup:

• docker is logged in on ops and simcore
• command: docker service create --mode global --name hello-world hello-world is executed on manager

Outcome:

• 1 docker pull requests were taken from anonymous user

Case 5. All machines are logged in. No --with-registry-auth option

Setup:

• docker is logged in on all machines
• command: docker service create --mode global --name hello-world hello-world is executed on manager

Outcome:

• 2 docker pull requests were taken from anonymous user

Case 6. All machines are [docker] logged out. With --with-registry-auth option. The image is present on all machines

Setup:

• command: docker service create --with-registry-auth --mode global --name hello-world hello-world is executed on manager

Outcome:

• 1 docker pull requests was taken from anonymous user

[sanderegg]

Docker log in per request vs Private registry as pull through cache to Dockerhub

Docker log in per request - need to be done per request (every time) - need to log in on every machine

Private registry as pull through cache to Dockerhub + once set up works automatically out-of-the-box with no overhead - OPS need to guarantee private registry uptime - OPS need to properly loadbalance and scale private registry - OPS need to maintain disk space of private registries - once registry is down no image can be pulled via regsitry - daemon.json needs to be configured on every machine

Decision Based on this comparison, I decide to go on with Docker log in per request approach. Even though I would rather have something working automatically out-of-the-box but the cost is too high

@mrnicegyu11, @sanderegg, let me know what are your thoughts on this matter

I would like to discuss these +/-. If the private registry is down that means anyway:

• nothing can run as all the services are only available there (s4l, isolve, etc...)
• OPS side, if the registry is down, then oSparc is anyway able to run nothing, so not sure this changes much,
• Costs side, it is probably an improvement, especially if the registry is setup with S3 VPC, and scaled on the regions we are using,
• number of pulls toward Dockerhub is down to a minimum

[YuryHrytsuk]

Docker log in per request vs Private registry as pull through cache to Dockerhub Docker log in per request - need to be done per request (every time) - need to log in on every machine Private registry as pull through cache to Dockerhub + once set up works automatically out-of-the-box with no overhead - OPS need to guarantee private registry uptime - OPS need to properly loadbalance and scale private registry - OPS need to maintain disk space of private registries - once registry is down no image can be pulled via regsitry - daemon.json needs to be configured on every machine Decision Based on this comparison, I decide to go on with Docker log in per request approach. Even though I would rather have something working automatically out-of-the-box but the cost is too high @mrnicegyu11, @sanderegg, let me know what are your thoughts on this matter

I would like to discuss these +/-. If the private registry is down that means anyway:

* nothing can run as all the services are only available there (s4l, isolve, etc...)

* OPS side, if the registry is down, then oSparc is anyway able to run nothing, so not sure this changes much,

* Costs side, it is probably an improvement, especially if the registry is setup with S3 VPC, and scaled on the regions we are using,

* number of pulls toward Dockerhub is down to a minimum

@mrnicegyu11, @sanderegg and I had a discussion and decided to try pull through registry approach. I will do a POC to check how / if it actually works

[YuryHrytsuk]

Docker registry pull through cache

Useful links:

• https://distribution.github.io/distribution/
• https://github.com/distribution/distribution/issues/3645
• https://shipyard.build/blog/how-to-docker-registry-pull-through-cache/

Insights:

• you cannot have a pull-through cache with auth (https://github.com/moby/moby/issues/30880)
  • @YuryHrytsuk tested locally and proved that this is the case

Conclusions

• Docker registry pull through cache can be made "private" via networking / firewalls
• It brings complexities which to me seems bigger than advantages
  • Complexity: it requires separate DNS or Node Port to maintain
  • Complexity: it requires cluster machines to stay private (not accessible to external users)
    • which is now not possible due to https://github.com/ITISFoundation/osparc-ops-environments/issues/718
  • Complexity: it requires one more registry since pull through cache registry does not support docker image push
  • Advantage: it does not require docker login and --with-registry-auth option to be executed

I am in favor of docker login + --with-registry-auth approach at this point of time. Once all our machines are private, we can consider it one more time.

UPD:

• There are potential problems with docker login approach: if we use it in every deployment, we may get pulled back by docker team. Also we should use separate user for each deployment for decoupling
• Since computational clusters are not part of swarm, we need to consider how to make pull through cache registry available to cluster keepers

[YuryHrytsuk]

Docker registry as pull through cache experiments

Context:

• daemon.json configured to use mirror registry

Outcomes:

• When mirror registry container is not running, docker pull as if there was no registry
• One can safely restart mirror registry without restarting docker. The mirror registry will be still used
• If credentials given to mirror registry are incorrect, docker will pull as if there was no registry
• If mirror registry is down, docker will still pull images

[YuryHrytsuk]

Blocked for AWS Deployments until OPS nodes are private https://github.com/ITISFoundation/osparc-ops-environments/issues/574