Use quay.io as default to mitigate dockerhub rate limit errors

[samip5]

Expected Behavior

You can freely download the calico container images on a fresh cluster

Current Behavior

Due to Docker Hub ratelimits, re-building a cluster and having many containers pull from Docker Hub might result in hitting rate-limits.

Possible Solution

Push Calico containers to Github Container Registry (ghcr) or quay.io

Context

https://docs.docker.com/docker-hub/download-rate-limit/

[onedr0p]

It would be nice to cover the following images on a separate registry.

docker.io/calico/typha
docker.io/calico/node
docker.io/calico/kube-controllers

[caseydavenport]

@onedr0p @samip5 all of those images are also available on quay.io already.

Another solution would be to download once to a local registry and serve the images locally - this is definitely the best solution for production clusters as it (1) reduces dependency on external infrastructure and (2) reduces ingress traffic to the cluster to download new images (once, rather than once per-node).

[caseydavenport]

I'm going to leave this open, since we should probably consider switching to quay.io as our default registry (again).

[onedr0p]

I did not know these were all on Quay, that's awesome! Seems like you need to modify some pipelines to push the multiarch images to quay and you'll be mostly set 👍🏼

[caseydavenport]

Seems like you need to modify some pipelines to push the multiarch images to quay and you'll be mostly set 👍🏼

I'll need to check if quay has finally started supporting multiarch images. We used to use quay as the default and swapped to dockerhub because at the time quay did not support multi-arch.

If that's changed that would be great, otherwise we have a tradeoff: ratelimit vs multi-arch support.

[caseydavenport]

haha, looks like you already found https://github.com/projectcalico/calico/issues/3520 and informed me that quay does now support multi-arch.

Yeah, we should probably start pushing multi-arch to quay and change our default.

[mkumatag]

any further progress made here?

[caseydavenport]

This might be all that is required: https://github.com/projectcalico/go-build/pull/351

[caseydavenport]

Multi-arch images are on quay now, changing the default still hasn't happened but for anyone finding this you should just be able to switch to images from quay.io/calico/

[macropin]

you should just be able to switch to images from quay.io/calico/

This may be daft question, but how is this done? Installing the tigera-operator using the helm chart doesn't appear to have this as a configuration option, only the following:

# Configuration for the tigera operator
tigeraOperator:
  image: tigera/operator
  version: v1.25.7
  registry: quay.io
calicoctl:
  image: docker.io/calico/ctl
  tag: v3.22.2

Is .Values.tigeraOperator.registry used as the registry for all the component images? Or can it only be configured after installation by editing the Installation custom resource.

[macropin]

Answering my own question above: The following values.yaml is required to use quay.io as the registry source for the operator.

installation:
  registry: quay.io/

[Cellebyte]

As you currently have an unpaid account. It would be beneficial to not break a lot of production clusters by either paying or announce a change of the default image registry soon.

https://blog.alexellis.io/docker-is-deleting-open-source-images/

Kind request what your current plan is regarding the distribution of container images of calico.

[Exagone313]

If you are using the tigera operator without helm, you can edit the Installation custom resource (installations.operator.tigera.io) to add:

# kubectl edit installations default
apiVersion: operator.tigera.io/v1
kind: Installation
# ...
spec:
  registry: "quay.io/"

Calico will then be redeployed using the Quay registry.

[Cellebyte]

If you are using the tigera operator without helm, you can edit the Installation custom resource (installations.operator.tigera.io) to add:

# kubectl edit installations default
apiVersion: operator.tigera.io/v1
kind: Installation
# ...
spec:
  registry: "quay.io/"

Calico will then be redeployed using the Quay registry.

@Exagone313 it is more about the people who never touch a running system ;) If they just announce changing to a different default registry that would also be fine for us. But using still a registry which would soon be wiped should not be considered user-friendly in my opinion. At least a statement what the next steps are from projectcalico/tigera would be beneficial so that organizations and deployments which depend on the current registry have enough time to switch registries or do nothing.

[onedr0p]

I agree the default should be Quay at this point moving forward.

https://github.com/docker/hub-feedback/issues/2314

[caseydavenport]

We should still switch to quay for sure, but in case anyone is worried about Calico's old images, I can confirm that we are not using a free tier plan. Additionally, Docker has announced that they are not going to remove old free team images anyway: https://www.docker.com/blog/no-longer-sunsetting-the-free-team-plan/

[gabegorelick]

Just got bit by this. I'm running the operator.

Failed to pull image "docker.io/calico/csi:v3.25.1": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/calico/csi:v3.25.1": failed to resolve reference "docker.io/calico/csi:v3.25.1": pulling from host registry-1.docker.io failed with status code [manifests v3.25.1]: 429 Too Many Requests

Switching to quay.io fixed it.

[PatrickLaabs]

Any plans on this to make quay.io the default address for pulling the calico images? 😄