search

operator-framework / operator-registry

Operator Registry runs in a Kubernetes or OpenShift cluster to provide operator catalog data to Operator Lifecycle Manager.
Apache License 2.0
211 stars 247 forks source link

Unable to use OpenShift Image Registry with opm index add 400 Bad Request #577

Open csantanapr opened 3 years ago

csantanapr commented 3 years ago

We are trying to build generic devops pipelines using OpenShift Pipelines (Tekton)

We want to have a devops pipeline that allows developer push their operator source code to git, and this triggers a tekton pipeline and for the development phase, we want to leverage the internal openshift image registry image-registry.openshift-image-registry.svc because this helps with latency when pushing/pulling also managing access per teams and projects using OpenShift RBAC each team can access their namespace imagestreams.

For authentication I don't know how to configure opm with the credentials of the service account pipeline (default for Tekton), when I use buildah I don't pass credentials buildah autodetects

In all opm commands below I configured $HOME/.docker/config.json with kubeadmin and $(oc whoami -t), notice I don't want to use pull-tool docker, I'm not passing a pull-tool since I'm just doing a generate Dockerfile and not going to build the image catalog image with opm, I will be using buildah in a later tekton task using the generated Dockerfile

We have the following line in our pipeline with opm

opm index add --skip-tls=true --bundles image-registry.openshift-image-registry.svc:5000/project1-dev/ansible-operator-bundle:0.0.3 --generate -d ./index.Dockerfile --debug true

The error logs 400 Bad Request

time="2021-02-10T00:18:05Z" level=warning msg="--skip-tls flag is set: this mode is insecure and meant for development purposes only."
time="2021-02-10T00:18:05Z" level=info msg="building the index" bundles="[image-registry.openshift-image-registry.svc:5000/project1-dev/ansible-operator-bundle:0.0.3]"
time="2021-02-10T00:18:05Z" level=debug msg=resolving host="image-registry.openshift-image-registry.svc:5000"
time="2021-02-10T00:18:05Z" level=debug msg="do request" host="image-registry.openshift-image-registry.svc:5000" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=opm/alpha request.method=HEAD url="http://image-registry.openshift-image-registry.svc:5000/v2/project1-dev/ansible-operator-bundle/manifests/0.0.3"
time="2021-02-10T00:18:05Z" level=debug msg="fetch response received" host="image-registry.openshift-image-registry.svc:5000" response.status="400 Bad Request" url="http://image-registry.openshift-image-registry.svc:5000/v2/project1-dev/ansible-operator-bundle/manifests/0.0.3"
time="2021-02-10T00:18:05Z" level=debug msg="unable to populate database: [error resolving name : unexpected status code [manifests 0.0.3]: 400 Bad Request, image \"image-registry.openshift-image-registry.svc:5000/project1-dev/ansible-operator-bundle:0.0.3\": not found]" bundles="[image-registry.openshift-image-registry.svc:5000/project1-dev/ansible-operator-bundle:0.0.3]"
time="2021-02-10T00:18:05Z" level=error msg="permissive mode disabled" bundles="[image-registry.openshift-image-registry.svc:5000/project1-dev/ansible-operator-bundle:0.0.3]" error="[error resolving name : unexpected status code [manifests 0.0.3]: 400 Bad Request, image \"image-registry.openshift-image-registry.svc:5000/project1-dev/ansible-operator-bundle:0.0.3\": not found]"
time="2021-02-10T00:18:05Z" level=debug msg="unable to add bundle to registry" bundles="[image-registry.openshift-image-registry.svc:5000/project1-dev/ansible-operator-bundle:0.0.3]" error="[error resolving name : unexpected status code [manifests 0.0.3]: 400 Bad Request, image \"image-registry.openshift-image-registry.svc:5000/project1-dev/ansible-operator-bundle:0.0.3\": not found]"
Error: [error resolving name : unexpected status code [manifests 0.0.3]: 400 Bad Request, image "image-registry.openshift-image-registry.svc:5000/project1-dev/ansible-operator-bundle:0.0.3": not found]
Usage:

I tried to debug the problem by exposing the registry and trying opm from my laptop, it doesn't work but I get a different problem

opm index add --skip-tls=true --bundles default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/project1-dev/ansible-operator-bundle:0.0.3 --generate -d ./index.Dockerfile --debug true

The error logs 503 Service Unavailable

WARN[0000] --skip-tls flag is set: this mode is insecure and meant for development purposes only.
INFO[0000] building the index                            bundles="[default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/project1-dev/ansible-operator-bundle:0.0.3]"
DEBU[0000] resolving                                     host=default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local
DEBU[0000] do request                                    host=default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=opm/alpha request.method=HEAD url="http://default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/v2/project1-dev/ansible-operator-bundle/manifests/0.0.3"
DEBU[0000] fetch response received                       host=default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local response.header.cache-control="private, max-age=0, no-cache, no-store" response.header.connection=close response.header.content-type=text/html response.header.pragma=no-cache response.status="503 Service Unavailable" url="http://default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/v2/project1-dev/ansible-operator-bundle/manifests/0.0.3"
DEBU[0000] unable to populate database: [error resolving name : unexpected status code [manifests 0.0.3]: 503 Service Unavailable, image "default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/project1-dev/ansible-operator-bundle:0.0.3": not found]  bundles="[default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/project1-dev/ansible-operator-bundle:0.0.3]"
ERRO[0000] permissive mode disabled                      bundles="[default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/project1-dev/ansible-operator-bundle:0.0.3]" error="[error resolving name : unexpected status code [manifests 0.0.3]: 503 Service Unavailable, image \"default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/project1-dev/ansible-operator-bundle:0.0.3\": not found]"
DEBU[0000] unable to add bundle to registry              bundles="[default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/project1-dev/ansible-operator-bundle:0.0.3]" error="[error resolving name : unexpected status code [manifests 0.0.3]: 503 Service Unavailable, image \"default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/project1-dev/ansible-operator-bundle:0.0.3\": not found]"
Error: [error resolving name : unexpected status code [manifests 0.0.3]: 503 Service Unavailable, image "default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/project1-dev/ansible-operator-bundle:0.0.3": not found]
Usage:

Then I updated the OpenShfit route to redirect HTTP to HTTPS

spec:
  host: default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local
  to:
    kind: Service
    name: image-registry
    weight: 100
  tls:
    termination: reencrypt
    insecureEdgeTerminationPolicy: Redirect
  wildcardPolicy: None

Then try again

opm index add --skip-tls=true --bundles default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/project1-dev/ansible-operator-bundle:0.0.3 --generate -d ./index.Dockerfile --debug true

The error log 401 Unauthorized

WARN[0000] --skip-tls flag is set: this mode is insecure and meant for development purposes only.
INFO[0000] building the index                            bundles="[default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/project1-dev/ansible-operator-bundle:0.0.3]"
DEBU[0000] resolving                                     host=default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local
DEBU[0000] do request                                    host=default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=opm/alpha request.method=HEAD url="http://default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/v2/project1-dev/ansible-operator-bundle/manifests/0.0.3"
DEBU[0000] fetch response received                       host=default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local response.header.content-length=179 response.header.content-type="application/json; charset=utf-8" response.header.date="Wed, 10 Feb 2021 03:30:29 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.set-cookie="34727b82525eb26a530629c5bf0ec2f2=45dcde97457968b194acdaa973d153f4; path=/; HttpOnly; Secure" response.header.www-authenticate="Bearer realm=\"https://default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/openshift/token\"" response.header.x-registry-supports-signatures=1 response.status="401 Unauthorized" url="http://default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/v2/project1-dev/ansible-operator-bundle/manifests/0.0.3"
DEBU[0000] Unauthorized                                  header="Bearer realm=\"https://default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/openshift/token\"" host=default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local
DEBU[0001] do request                                    host=default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=opm/alpha request.method=HEAD url="http://default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/v2/project1-dev/ansible-operator-bundle/manifests/0.0.3"
DEBU[0001] fetch response received                       host=default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local response.header.content-length=179 response.header.content-type="application/json; charset=utf-8" response.header.date="Wed, 10 Feb 2021 03:30:30 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.set-cookie="34727b82525eb26a530629c5bf0ec2f2=45dcde97457968b194acdaa973d153f4; path=/; HttpOnly; Secure" response.header.www-authenticate="Bearer realm=\"https://default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/openshift/token\"" response.header.x-registry-supports-signatures=1 response.status="401 Unauthorized" url="http://default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/v2/project1-dev/ansible-operator-bundle/manifests/0.0.3"
DEBU[0001] Unauthorized                                  header="Bearer realm=\"https://default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/openshift/token\"" host=default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local
DEBU[0001] do request                                    host=default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=opm/alpha request.method=HEAD url="http://default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/v2/project1-dev/ansible-operator-bundle/manifests/0.0.3"
DEBU[0002] fetch response received                       host=default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local response.header.content-length=179 response.header.content-type="application/json; charset=utf-8" response.header.date="Wed, 10 Feb 2021 03:30:30 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.set-cookie="34727b82525eb26a530629c5bf0ec2f2=45dcde97457968b194acdaa973d153f4; path=/; HttpOnly; Secure" response.header.www-authenticate="Bearer realm=\"https://default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/openshift/token\"" response.header.x-registry-supports-signatures=1 response.status="401 Unauthorized" url="http://default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/v2/project1-dev/ansible-operator-bundle/manifests/0.0.3"
DEBU[0002] Unauthorized                                  header="Bearer realm=\"https://default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/openshift/token\"" host=default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local
DEBU[0002] do request                                    host=default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=opm/alpha request.method=HEAD url="http://default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/v2/project1-dev/ansible-operator-bundle/manifests/0.0.3"
DEBU[0002] fetch response received                       host=default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local response.header.content-length=179 response.header.content-type="application/json; charset=utf-8" response.header.date="Wed, 10 Feb 2021 03:30:31 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.set-cookie="34727b82525eb26a530629c5bf0ec2f2=45dcde97457968b194acdaa973d153f4; path=/; HttpOnly; Secure" response.header.www-authenticate="Bearer realm=\"https://default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/openshift/token\"" response.header.x-registry-supports-signatures=1 response.status="401 Unauthorized" url="http://default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/v2/project1-dev/ansible-operator-bundle/manifests/0.0.3"
DEBU[0002] Unauthorized                                  header="Bearer realm=\"https://default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/openshift/token\"" host=default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local
DEBU[0002] do request                                    host=default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=opm/alpha request.method=HEAD url="http://default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/v2/project1-dev/ansible-operator-bundle/manifests/0.0.3"
DEBU[0002] fetch response received                       host=default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local response.header.content-length=179 response.header.content-type="application/json; charset=utf-8" response.header.date="Wed, 10 Feb 2021 03:30:31 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.set-cookie="34727b82525eb26a530629c5bf0ec2f2=45dcde97457968b194acdaa973d153f4; path=/; HttpOnly; Secure" response.header.www-authenticate="Bearer realm=\"https://default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/openshift/token\"" response.header.x-registry-supports-signatures=1 response.status="401 Unauthorized" url="http://default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/v2/project1-dev/ansible-operator-bundle/manifests/0.0.3"
DEBU[0002] Unauthorized                                  header="Bearer realm=\"https://default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/openshift/token\"" host=default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local
DEBU[0002] do request                                    host=default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=opm/alpha request.method=HEAD url="http://default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/v2/project1-dev/ansible-operator-bundle/manifests/0.0.3"
DEBU[0003] fetch response received                       host=default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local response.header.content-length=179 response.header.content-type="application/json; charset=utf-8" response.header.date="Wed, 10 Feb 2021 03:30:31 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.set-cookie="34727b82525eb26a530629c5bf0ec2f2=45dcde97457968b194acdaa973d153f4; path=/; HttpOnly; Secure" response.header.www-authenticate="Bearer realm=\"https://default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/openshift/token\"" response.header.x-registry-supports-signatures=1 response.status="401 Unauthorized" url="http://default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/v2/project1-dev/ansible-operator-bundle/manifests/0.0.3"
DEBU[0003] unable to populate database: [error resolving name : unexpected status code [manifests 0.0.3]: 401 Unauthorized, image "default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/project1-dev/ansible-operator-bundle:0.0.3": not found]  bundles="[default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/project1-dev/ansible-operator-bundle:0.0.3]"
ERRO[0003] permissive mode disabled                      bundles="[default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/project1-dev/ansible-operator-bundle:0.0.3]" error="[error resolving name : unexpected status code [manifests 0.0.3]: 401 Unauthorized, image \"default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/project1-dev/ansible-operator-bundle:0.0.3\": not found]"
DEBU[0003] unable to add bundle to registry              bundles="[default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/project1-dev/ansible-operator-bundle:0.0.3]" error="[error resolving name : unexpected status code [manifests 0.0.3]: 401 Unauthorized, image \"default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/project1-dev/ansible-operator-bundle:0.0.3\": not found]"
Error: [error resolving name : unexpected status code [manifests 0.0.3]: 401 Unauthorized, image "default-route-openshift-image-registry.apps.csan-vm3.ocp.csplab.local/project1-dev/ansible-operator-bundle:0.0.3": not found]
Usage:
codihuston commented 2 years ago

Any update on this @csantanapr? I am in a similar position (not an http 401), but when I apply the subscription, the image path to the operator is referencing the external image url (not the internal one). Not sure if this is related, but I am pretty sure that this stems from building and pushing the index image requiring me to use the external url (as opposed to the internal image-registry.openshift-image-registry.svc:5000/my-project/my-operator-index:tag path)

pkhanpara commented 2 years ago

AFAIK, there is no need to redirect HTTP to HTTPS. Removing --skip-tls=true worked for me. HTH