search

minio / operator

Simple Kubernetes Operator for MinIO clusters :computer:
https://min.io/docs/minio/kubernetes/upstream/index.html
GNU Affero General Public License v3.0
1.18k stars 449 forks source link

Operator gets rate-limited by Docker Hub when using non-standard image #1092

Closed funkypenguin closed 2 years ago

funkypenguin commented 2 years ago

Expected Behavior

We mirror all images which we use in our cluster, from the official registries into our own repository, so that we can (a) scan them consistently, and (b) whitelist the registry using an admission controller. For this reason, we're using (for example) docker.io/myorg/minio as the image base.

Minio-operator seems to poll Docker Hub to look for available upgrades, but it does so so frequently that it hits the rate limit (below), and logs a message about unable to locate auth config registry context index.docker.io

Current Behavior

Logs as described above:

I0412 22:34:36.429681       1 artifacts.go:134] unable to locate auth config registry context index.docker.io
E0412 22:34:36.839650       1 main-controller.go:579] error syncing 'preview-database-pr-236/minio': GET https://index.docker.io/v2/myorg/minio/manifests/RELEASE.2022-03-05T06-32-39Z: TOOMANYREQUESTS: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

Possible Solution

A few ideas ...

  1. Allow the user to specify credentials used to authenticate to the registry for pulls (we use a public repository currently, but this would also help users who use private repos)
  2. Allow the user to modify the frequency with which minio-operator checks for updates, or to disable this feature entirely
  3. Allow the user to specify a registry mirror URL to be used for this check (in our case, we actually do store all images in a local registry which is not subject to rate limiting)

Steps to Reproduce (for bugs)

  1. Pull the latest minio/minio image from docker hub, re-tag it as /minio, and push it back to Docker Hub
  2. Create a minio-operator / tenant combination using your image (presumably the official minio/minio image is exempt from ratelimits)
  3. Observe the logs in the operator as the rate-limit is reached

Context

As described above, we regularly scan and update images from multiple sources into our registry/repo, and configure our cluster to only run images from these trusted repositories. In our case, we'd be 100% OK with disabling checks for image updates, since triggering these is a manual process anyway.

Your Environment

harshavardhana commented 2 years ago

Use image pull secrets and pull images by authenticating with docker hub - or use quay.io/minio/minio

funkypenguin commented 2 years ago

Is it possible to disable the checking for new versions? It seems very aggressive (below), retrying every 20s or so, so even though I'm authenticated against Docker Hub, I hit my rate limit on a fresh instance within about 10 minutes...

I0418 01:51:35.942409       1 main-controller.go:1062] Tenant 'minio' MinIO is already running the most recent version of 2022-04-12T06:55:35Z
I0418 01:51:43.486318       1 main-controller.go:1062] Tenant 'minio' MinIO is already running the most recent version of 2022-04-12T06:55:35Z
I0418 01:51:50.325559       1 main-controller.go:1062] Tenant 'minio' MinIO is already running the most recent version of 2022-04-12T06:55:35Z
I0418 01:51:58.078473       1 main-controller.go:1062] Tenant 'minio' MinIO is already running the most recent version of 2022-04-12T06:55:35Z
I0418 01:52:06.651582       1 main-controller.go:1062] Tenant 'minio' MinIO is already running the most recent version of 2022-04-12T06:55:35Z
E0418 01:52:07.987733       1 main-controller.go:579] error syncing 'minio/minio': GET https://index.docker.io/v2/myorg/minio/manifests/RELEASE.2022-04-12T06-55-35Z: TOOMANYREQUESTS: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
E0418 01:52:23.182750       1 main-controller.go:579] error syncing 'minio/minio': GET https://index.docker.io/v2/myorg/minio/manifests/RELEASE.2022-04-12T06-55-35Z: TOOMANYREQUESTS: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
E0418 01:52:42.875705       1 main-controller.go:579] error syncing 'minio/minio': GET https://index.docker.io/v2/myorg/minio/manifests/RELEASE.2022-04-12T06-55-35Z: TOOMANYREQUESTS: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
harshavardhana commented 2 years ago

Is it possible to disable the checking for new versions? It seems very aggressive (below), retrying every 20s or so, so even though I'm authenticated against Docker Hub, I hit my rate limit on a fresh instance within about 10 minutes...

I0418 01:51:35.942409       1 main-controller.go:1062] Tenant 'minio' MinIO is already running the most recent version of 2022-04-12T06:55:35Z
I0418 01:51:43.486318       1 main-controller.go:1062] Tenant 'minio' MinIO is already running the most recent version of 2022-04-12T06:55:35Z
I0418 01:51:50.325559       1 main-controller.go:1062] Tenant 'minio' MinIO is already running the most recent version of 2022-04-12T06:55:35Z
I0418 01:51:58.078473       1 main-controller.go:1062] Tenant 'minio' MinIO is already running the most recent version of 2022-04-12T06:55:35Z
I0418 01:52:06.651582       1 main-controller.go:1062] Tenant 'minio' MinIO is already running the most recent version of 2022-04-12T06:55:35Z
E0418 01:52:07.987733       1 main-controller.go:579] error syncing 'minio/minio': GET https://index.docker.io/v2/myorg/minio/manifests/RELEASE.2022-04-12T06-55-35Z: TOOMANYREQUESTS: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
E0418 01:52:23.182750       1 main-controller.go:579] error syncing 'minio/minio': GET https://index.docker.io/v2/myorg/minio/manifests/RELEASE.2022-04-12T06-55-35Z: TOOMANYREQUESTS: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
E0418 01:52:42.875705       1 main-controller.go:579] error syncing 'minio/minio': GET https://index.docker.io/v2/myorg/minio/manifests/RELEASE.2022-04-12T06-55-35Z: TOOMANYREQUESTS: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

This is because you asked for upgrade when it is already upgraded. The declarative nature here is trying to push the changes that are relevant and what was asked..

harshavardhana commented 2 years ago

@dvaldivia we should treat the condition when the server is already upgraded as good condition and avoid returning error.

This would avoid pulling in the images or hitting docker hub.

funkypenguin commented 2 years ago

This is because you asked for upgrade when it is already upgraded. The declarative nature here is trying to push the changes that are relevant and what was asked..

Sorry, missed this response here. How did I ask for an upgrade? (I didn't intend to...)

D