Not getting images from mirror-registry

[eb4x]

What happened: The cronjobs fetching fedora/centos images time out in an air-gapped cluster.

I0805 13:40:08.423084 1 registry-datasource.go:176] Copying proxy certs
2024/08/05 13:40:08 Ignore common certificate dir: open /proxycerts/: no such file or directory
I0805 13:40:08.423157 1 transport.go:228] Inspecting image from 'docker://quay.io/containerdisks/fedora:latest'
E0805 13:41:08.425189 1 transport.go:78] Could not create image reference: pinging container registry quay.io: Get "https://quay.io/v2/": dial tcp 34.206.201.197:443: i/o timeout
2024/08/05 13:41:08 Failed to get image digest: Could not create image reference: pinging container registry quay.io: Get "https://quay.io/v2/": dial tcp 34.206.201.197:443: i/o timeout

What you expected to happen: Getting the image from a mirror via the /etc/containers/registries.conf.d/99-mirror-registries.conf configuration on each node.

[[registry]]
prefix="quay.io"
location="mirror-registry.openshift-utv.uio.no:8443/mirrors/quay.io"

Additional context:

I have a baremetal OKD 4.14 SCOS in an semi-air-gapped environment. I push images to a minimal mirror-registry (quay) separate from the cluster, and all nodes have a configuration (see above snippet) that forwards the most common registry-urls to a location in mirror-registry.

Environment:

• CDI version (use kubectl get deployments cdi-deployment -o yaml): 1.10.7
• Kubernetes version (use kubectl version): v1.27.1-3507+28ed2d70d51caa-dirty
• DV specification: N/A
• Cloud provider or hardware configuration: OKD/Baremetal
• OS (e.g. from /etc/os-release): CentOS Stream CoreOS 414.9.202401231453-0
• Kernel (e.g. uname -a): 5.14.0-410.el9.x86_64

[aglitke]

@arnongilboa could you take a look at this? It seems we should be supporting this scenario. @eb4x Are you using HCO to deploy the kubevirt components are are you doing a more custom setup? Make sure that you have configured your DataImportCron objects to point to the correct registry. From the error message it looks like you are still trying to fetch from quay.

[arnongilboa]

@aglitke we should definitely support this scenario. @eb4x As mentioned, the DataImportCrons should point urls in the mirror-registry. See here. If you are using OpenShift (with HCO) you may follow this.

[eb4x]

@eb4x Are you using HCO to deploy the kubevirt components are are you doing a more custom setup?

Yep, I'm using the HCO to deploy.

Make sure that you have configured your DataImportCron objects to point to the correct registry. From the error message it looks like you are still trying to fetch from quay.

I'll look into this further, I might have missed something. The idea behind our registry overrides is that we don't have to change where we're pulling from. The worker nodes should know to pull from the mirror-registry.

It works for resources specified in kubernetes. (deployments, daemonsets, pods, etc.) But maybe not from within a running container which is kinda what's happening here?