Open edsantiago opened 2 months ago
For reference this is the used registries.conf: https://github.com/containers/podman/blob/main/test/registries-cached.conf
containers-registries.conf(5) says:
Redirection and mirrors are currently processed only when reading a single image, not when pushing to a registry nor when doing any other kind of lookup/search on a on a registry. This may change in the future.
Given --all-tags must lookup all tags first then pull image the tag list still reads from quay.io but I don't know why it is that way. It certainly breaks the usage of this in restricted environments.
cc @mtrmac
Also trying locally with debug logs shows this very clearly
DEBU[0000] GET https://quay.io/v2/
DEBU[0000] Ping https://quay.io/v2/ status 401
DEBU[0000] GET https://quay.io/v2/auth?scope=repository%3Alibpod%2Ftestdigest_v2s2%3Apull&service=quay.io
DEBU[0001] Increasing token expiration to: 60 seconds
DEBU[0001] GET https://quay.io/v2/libpod/testdigest_v2s2/tags/list
DEBU[0001] Looking up image "quay.io/libpod/testdigest_v2s2:20200210" in local containers storage
DEBU[0001] Normalized platform linux/amd64 to {amd64 linux [] }
DEBU[0001] Trying "quay.io/libpod/testdigest_v2s2:20200210" ...
DEBU[0001] reference "[overlay@/home/pholzing/.local/share/containers/storage+/run/user/1000/containers]quay.io/libpod/testdigest_v2s2:20200210" does not resolve to an image ID
DEBU[0001] Trying "quay.io/libpod/testdigest_v2s2:20200210" ...
DEBU[0001] reference "[overlay@/home/pholzing/.local/share/containers/storage+/run/user/1000/containers]quay.io/libpod/testdigest_v2s2:20200210" does not resolve to an image ID
DEBU[0001] Trying "quay.io/libpod/testdigest_v2s2:20200210" ...
DEBU[0001] Normalized platform linux/amd64 to {amd64 linux [] }
DEBU[0001] Attempting to pull candidate quay.io/libpod/testdigest_v2s2:20200210 for quay.io/libpod/testdigest_v2s2:20200210
DEBU[0001] parsed reference into "[overlay@/home/pholzing/.local/share/containers/storage+/run/user/1000/containers]quay.io/libpod/testdigest_v2s2:20200210"
Trying to pull quay.io/libpod/testdigest_v2s2:20200210...
DEBU[0001] Copying source image //quay.io/libpod/testdigest_v2s2:20200210 to destination image [overlay@/home/pholzing/.local/share/containers/storage+/run/user/1000/containers]quay.io/libpod/testdigest_v2s2:20200210
DEBU[0001] Using registries.d directory /etc/containers/registries.d
DEBU[0001] Trying to access "127.0.0.1:60333/libpod/testdigest_v2s2:20200210"
Yes, mirrors are not used for listing tags.
It’s … consistent? The mirror code is designed to work well with partial mirrors, where important images are mirrored and pulling the others falls back to the primary. But in such an environment, listing tags from the mirror returns unexpected results.
Really, this is more of an implementation accident (and delivering a minimal viable mirroring implementation) than a well-considered careful decision.
We could add the feature, and an option to registries.conf
to opt into listing tags from mirrors. But then again, I don’t know who should ever use podman pull --all-tags
. Other users of “list tags” are likely to be mirroring tools and the like who care about contacting the primary vs. a mirror, and know the details of the setup; such users can trigger a tag listing of the mirror explicitly, if that’s what they want.
For the test, would it be practical to run pul --all-tags
directly from the mirror location? It does affect what is being tested, I don’t know whether that’s acceptable.
For the test, would it be practical to run
pull --all-tags
directly from the mirror location?
Tentative yes, but there seems to be a gap in the passing of --tls-verify
:
$ bin/podman pull --tls-verify=false --all-tags 127.0.0.1:60333/libpod/testdigest_v2s2
Error: pinging container registry 127.0.0.1:60333: Get "https://127.0.0.1:60333/v2/": http: server gave HTTP response to HTTPS client
$ ^-a
...works fine...
(Side note: for reasons I can't explain, my brain was convinced that search
would not work on a local registry. It does! I've made myself a TODO subitem on my protect-us-from-quay-outages list)
For the test, would it be practical to run pul --all-tags directly from the mirror location? It does affect what is being tested, I don’t know whether that’s acceptable.
The challenge here is that we must keep the tests working with and without the local registry setup so we add some complexity into the test to allow that. That said given quay.io flakes are bad enough it seems certainly worth it to do.
We already have a mechanism for conditionalizing e2e tests such that they work in and out of CI: https://github.com/containers/podman/blob/734c4b98d4fa006521a10515c0e9d3f4281bc2da/test/e2e/pull_test.go#L25-L32
I hate it, really hate the hardcoding and special-casing, hate the fact that some code paths go untested. I choose to accept this grossness for now as a pragmatic tradeoff. I hope some day we can reevaluate.
For the test, would it be practical to run
pull --all-tags
directly from the mirror location?Tentative yes, but there seems to be a gap in the passing of
--tls-verify
:
Huh. From a quick read of the code (without testing) this seems to be wired correctly, I don’t immediately see where the information is lost.
(Side note: for reasons I can't explain, my brain was convinced that
search
would not work on a local registry. It does! )
There are two APIs: /v1/search
, which does a server-side search (possible at scale) and is not implemented by docker/distribution at all; and /v2/_catalog
, which lists everything and searches client-side (unreasonable for large-scale hosted registries) and is implemented by docker/distribution.
The two are basically distinct code paths, testing one does not test the other. (And it’s up to the registry being contacted which one is used; we prefer /v1/search
.
A friendly reminder that this issue had no activity for 30 days.
@edsantiago what do you want to do with this one?
This is beyond my ability to fix, unless we decide to go with a documentation-only note. @Luap99 @mtrmac
I would be fine with just documenting this.
If --tls-verify
is not working, we should definitely fix that part.
podman pull --all-tags
CI test is flaking today due to the quay outage.Question: is there any way to force
--all-tags
to respect the aliases inregistries.conf
?Issue: docs need improvement because "docker.io" is misleading and wrong and confusing: