Missing documentation on image registry name/key expected by `opm` for DockerHub

[msugakov]

When opm registry add command is invoked (in quay.io/operator-framework/upstream-opm-builder:latest image, which is in turn invoked from operator-sdk run bundle, but these are not significant details), it tries to read $HOME/.docker/config.json file looking for credentials to authenticate with private container registry.

In my case registry is Docker Hub and config.json is configured with "auths" and without credentials helper (this is kubernetes.io/dockerconfigjson secret mounted in a pod), e.g.

{"auths":{"index.docker.io":{"username":"my-login","password":"Redacted123","email":"misha@mailme.com","auth":"bAsE64HeReRedaCtedAls0=="}}}

In this case opm registry add command fails with the output like this:

Details

``` opm registry add --debug -d /database/index.db -b docker.io/stackrox/stackrox-operator-bundle:v3.61.0-97-g97aef8009c -dirty --mode=semver --skip-tls=false INFO[0000] adding to the registry bundles="[docker.io/stackrox/stackrox-operator-bundle:v3.61.0-97-g97aef8009c-dirty]" DEBU[0000] resolving host=registry-1.docker.io DEBU[0000] do request host=registry-1.docker.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=opm/alpha request.method=HEAD url="https://registry-1.docker.io/v2/stackrox/stackrox-operator-bundle/manifests/v3.61.0-97-g97aef8009c-dirty" DEBU[0000] fetch response received host=registry-1.docker.io response.header.content-length=176 response.header.content-type=application/json response.header.date="Tue, 15 Jun 2021 21:15:44 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.strict-transport-security="max-age=31536000" response.header.www-authenticate="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\",scope=\"repository:stackrox/stackrox-operator-bundle:pull\"" response.status="401 Unauthorized" url="https://registry-1.docker.io/v2/stackrox/stackrox-operator-bundle/manifests/v3.61.0-97-g97aef8009c-dirty" DEBU[0000] Unauthorized header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\",scope=\"repository:stackrox/stackrox-operator-bundle:pull\"" host=registry-1.docker.io DEBU[0000] no scope specified for token auth challenge host=registry-1.docker.io DEBU[0000] do request host=registry-1.docker.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=opm/alpha request.method=HEAD url="https://registry-1.docker.io/v2/stackrox/stackrox-operator-bundle/manifests/v3.61.0-97-g97aef8009c-dirty" DEBU[0001] fetch response received host=registry-1.docker.io response.header.content-length=176 response.header.content-type=application/json response.header.date="Tue, 15 Jun 2021 21:15:45 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.strict-transport-security="max-age=31536000" response.header.www-authenticate="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\",scope=\"repository:stackrox/stackrox-operator-bundle:pull\",error=\"insufficient_scope\"" response.status="401 Unauthorized" url="https://registry-1.docker.io/v2/stackrox/stackrox-operator-bundle/manifests/v3.61.0-97-g97aef8009c-dirty" DEBU[0001] Unauthorized header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\",scope=\"repository:stackrox/stackrox-operator-bundle:pull\",error=\"insufficient_scope\"" host=registry-1.docker.io DEBU[0001] unable to populate database: [error resolving name : pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed, image "docker.io/stackrox/stackrox-operator-bundle:v3.61.0-97-g97aef8009c-dirty": not found] bundles="[docker.io/stackrox/stackrox-operator-bundle:v3.61.0-97-g97aef8009c-dirty]" ERRO[0001] permissive mode disabled bundles="[docker.io/stackrox/stackrox-operator-bundle:v3.61.0-97-g97aef8009c-dirty]" error="[error resolving name : pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed, image \"docker.io/stackrox/stackrox-operator-bundle:v3.61.0-97-g97aef8009c-dirty\": not found]" Error: [error resolving name : pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed, image "docker.io/stackrox/stackrox-operator-bundle:v3.61.0-97-g97aef8009c-dirty": not found] Usage: opm registry add [flags] Flags: -b, --bundle-images strings comma separated list of links to bundle image --ca-file string the root certificates to use when --container-tool=none; see docker/podman docs for certificate loading instructions -c, --container-tool string tool to interact with container images (save, build, etc.). One of: [none, docker, podman] (default "none") -d, --database string relative path to database file (default "bundles.db") --debug enable debug logging -h, --help help for add --mode string graph update mode that defines how channel graphs are updated. One of: [replaces, semver, semver-skippatch] (default "replaces") --permissive allow registry load errors Global Flags: --skip-tls skip TLS certificate verification for container image registries while pulling bundles or index ```

opm does not recognize either of the following as registry/host name:

• docker.io
• index.docker.io
• registry-1.docker.io

Instead, resolveHostname() function https://github.com/operator-framework/operator-registry/blob/da57f13b6d89bb3d905387f53ce14d905da6d9d2/pkg/image/containerdregistry/resolver.go#L105-L112 is invoked in such a way https://github.com/operator-framework/operator-registry/blob/da57f13b6d89bb3d905387f53ce14d905da6d9d2/pkg/image/containerdregistry/resolver.go#L70-L73 that its returned value, which is https://index.docker.io/v1/, must be a key in config.json's "auths".

Obviously, if config.json is modified so that https://index.docker.io/v1/ becomes the key, opm registry add works without an issue.

{"auths":{"https://index.docker.io/v1/":{"username":"my-login","password":"Redacted123","email":"misha@mailme.com","auth":"bAsE64HeReRedaCtedAls0=="}}}

I find this behavior surprising provided that opm documentation and built-in usage help does not mention anything about use of https://index.docker.io/v1/ as key for configuring Docker Hub credentials.

Could you kindly extend documentation to mention this behavior or extend opm in a way that it is more flexibly accepts Docker Hub hostnames?

[exdx]

The expected behavior should be that opm points to a default docker registry even if that's empty.

We should either make these registries configurable or document them more explicitly, and this issue can serve as a documentation itself.

[Ozfer]

Years later and still an issue. Still no documentation on the RedHat page and you have to run into the issue and then google it to find the solution. Why would it look to the default docker file and not where podman creates it is beyond me.

[joelanford]

@Ozfer this repo is a community-run project, and we unfortunately don't always have time to stay on top of all of the issues here. We do however have a Kubernetes slack channel and a weekly working group meeting where anyone from the community can bring discussion topics to the maintainers. We are happy to engage! Details here: https://github.com/operator-framework/community?tab=readme-ov-file#operator-lifecycle-manager-working-group

Do you know if there is a bug submitted to Red Hat about this? If not, that would probably explain why there are no Red Hat docs about it.

What version of opm are you using? I made some changes to the underlying resolver implementation in #1165 (release 1.33.0) to use a different credential loader that includes a bunch of improvements (including loading from podman's default config location).

Could you please check to see if this particular issue is resolved in version 1.33 or higher?