search

kopeio / etcd-manager

operator for etcd: moved to https://github.com/kubernetes-sigs/etcdadm
Apache License 2.0
164 stars 45 forks source link

DockerHub Rate Limit #404

Closed alexfrancavilla closed 3 years ago

alexfrancavilla commented 3 years ago

Hi, one of our Kops clusters was unable to recover from a minor config change, after debugging we found out that our master nodes were crashing because etcd was missing. The Docker Hub Rate Limit, which was introduced on November 20 last year, prevented our nodes to pull the etcd-manager image. Since then, more and more Kubernetes related images were moved to other registries or offer alternatives to circumvent these issues.

Is there any chance in having an alternative registry available?

In our organization we have an internal registry which also acts as a pull-through cache for Docker Hub, which includes credentials for a paid account. However in kops specifically I haven't found a way to change the pull URL, of course this not so much related to this project, I might create a separate issue on Kops for that.

For context, here is the log entry from systemctl -u kubelet.service of a failing master:

ErrImagePull: "rpc error: code = Unknown desc = Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit"
olemarkus commented 3 years ago

Have a look at https://kops.sigs.k8s.io/cluster_spec/#registry-mirrors

alexfrancavilla commented 3 years ago

@olemarkus Great, thanks for the pointer, that's basically what I was searching for :)

Feel free to close this issue if you don't see any other motivation to move away from Docker Hub in general.

olemarkus commented 3 years ago

Good to know!

/close