Mirrors are ignored when registry is blocked

[dcraigmv]

We are in a disconnected environment running a mirror of quay.io and to stop openshift from attempting to pull from quay.io the registry is marked as blocked but with mirrors:

[[registry]]
prefix = ""
location = "quay.io"
blocked = true

[[registry.mirror]]
location = "registry.internal.dev"

docker_client.go#L255 checks only that the main registry is blocked and returns an error.

I would expect it to then go on to check each mirror.

[mtrmac]

Thanks for reaching out.

What is the precise expected vs. actual behavior? Including the reported error message and possibly debug logs., please.

Are you saying that you do want the mirror to be reached, but that the blocked = true check on the listed line applies to the mirror? That’s not how the code is supposed to work (newDockerClient is called with the mirror’s reference), so if you do see that, that would be a bug and I’d like to see logs and details

[dcraigmv]

The actual behaviour is that the pod fails to pull the image with an event: Failed to pull image: registry quay.io/..... is blocked in /etc/containers/registries.conf or ..... This is using the registries.conf above (set via an ImageDigestMirrorSet)

I would expect that if a registry is configured but blocked and it also has mirrors configured, it should then check each mirror and use the first unblocked mirror it finds. If the registry is blocked and has no mirrors configured then of course it should err as it is currently.

It might be as simple as if reg.Blocked && reg.Mirrors == nil { but i'm not familiar with this codebase.

I have attempted to remove blocked = true for the quay.io registry in /etc/containers/registries.conf but that then throws up a dns error or connection refused if i try to sinkhole it. I have not tried to redirect quay.io to registry.internal.dev but i predict that would then lead to authentication errors.

[mtrmac]

I’m afraid without specific, debug logs and version numbers I’m still reduced to guessing.

Given that limitation, the error does not include Mirrors also failed (added somewhat recently), so my best guess is that the mirrors are not blocked, they are not processed at all. Do you happen to be using ImageContentSourcePolicy and a tag (not digested) reference for the image?

[dcraigmv]

I've attempted to reproduce this outside of the disconnected cluster using skopeo 1.16.0 and containers/image/v5 v5.32.0

sh-3.2$ cat /opt/homebrew/etc/containers/registries.conf
[[registry]]
prefix = ""
location = "quay.io"
blocked = true

[[registry.mirror]]
location = "registry.docker.io"

sh-3.2$ skopeo --debug list-tags docker://quay.io/openshift-release-dev/ocp-release
DEBU[0000] Using registries.d directory /opt/homebrew/etc/containers/registries.d 
DEBU[0000] Loading registries configuration "/opt/homebrew/etc/containers/registries.conf" 
DEBU[0000] No credentials matching quay.io/openshift-release-dev/ocp-release found in /Users/dcraigmv/.config/containers/auth.json 
DEBU[0000] No credentials matching quay.io/openshift-release-dev/ocp-release found in /Users/dcraigmv/.config/containers/auth.json 
DEBU[0000] Found an empty credential entry "quay.io" in "/Users/dcraigmv/.docker/config.json" (an unhandled credential helper marker?), moving on 
DEBU[0000] No credentials matching quay.io/openshift-release-dev/ocp-release found in /Users/dcraigmv/.dockercfg 
DEBU[0000] No credentials for quay.io/openshift-release-dev/ocp-release found 
DEBU[0000]  No signature storage configuration found for quay.io/openshift-release-dev/ocp-release:latest, using built-in default file:///Users/dcraigmv/.local/share/containers/sigstore 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/quay.io 
FATA[0000] Error listing repository tags: failed to create client: registry quay.io is blocked in /opt/homebrew/etc/containers/registries.conf or /Users/dcraigmv/.config/containers/registries.conf.d 
sh-3.2$

I would have expected that to at least try to connect to registry.docker.io/openshift-release-dev/ocp-release

[dcraigmv]

To add some context behind the registries.conf used, we have configured Openshift to NeverContactSource so it is adding blocked = true because that registry will never be accessible, so the mirror should always be used instead.

[mtrmac]

Thanks, that is much simpler: list-tags is not wired to use mirrors at all. Per container-registries.conf(5):

Redirection and mirrors are currently processed only when reading a single image, not when pushing to a registry nor when doing any other kind of lookup/search on a on a registry. This may change in the future.

But that is not related to how CRI-O implements image pulls in OpenShift (if that’s what the workload is).

Using the same config file with skopeo --debug … copy docker://quay.io/openshift-release-dev/ocp-release dir:t fails with:

initializing source docker://quay.io/openshift-release-dev/ocp-release:latest: (Mirrors also failed: [registry.docker.io/openshift-release-dev/ocp-release:latest: pinging container registry registry.docker.io: received unexpected HTTP status: 503 Service Unavailable]): quay.io/openshift-release-dev/ocp-release:latest: registry quay.io is blocked in …

and that’s the general shape of error I’d expect to see when mirrors are configured and the primary is blocked.

[dcraigmv]

In that case i don't understand what openshift is doing on the node, it's trying to pull the image via tag: openshift4/ose-kube-rbac-proxy:6360a467 which we have on our internal registry.

[mtrmac]

Actual configuration inputs? Generated registries.conf on the node? Version numbers? CRI-O debug-level logs?

[dcraigmv]

Unfortunately no not possible, the generated registries.conf is nearly identical to the above with hosts changed, i'm not sure how to get the version number of the library used out of openshift, any advice?

[mtrmac]

Gather those inputs all the same, and review them yourself, I expect.

[mtrmac]

(The c/image version number can be extracted determined from the CRI-O version number / commit, but because the code is vendored, just reading the CRI-O relevant source code tree is ~sufficient.)

[mtrmac]

I don’t know that we can help without more data. Please reopen if there are specific questions.