insecure-registry flag is not used when pushing referenced images

[prakashmirji]

What is your question? porter publish command fails after adding images section into poter.yaml

What have you tried already? Added below part to the porter.yaml

Note: registry URL: my-registry.com is masked one

images:
  assets:
    description: "A simple asset image"
    imageType: "docker"
    repository: "my-registry.com/develop/prakash/assets-nginx"
    digest: "sha256:9123250178a23334761946ac478aa56ec28f3f6402a491bb154fa13c724482b9"

Then tried to publish the bundle using below command

porter publish --registry my-registry.com/develop/prakash/ --insecure-registry --debug

I see below failure messages, not sure how to troubleshoot this. Appreciate pointers to troubleshoot it

DEBUG name: exec DEBUG pkgDir: /Users/prakash/.porter/mixins/exec DEBUG file:
DEBUG stdin:

/Users/prakash/.porter/mixins/exec/exec version --output json --debug DEBUG name: kubernetes DEBUG pkgDir: /Users/prakash/.porter/mixins/kubernetes DEBUG file:
DEBUG stdin:

/Users/prakash/.porter/mixins/kubernetes/kubernetes version --output json --debug Pushing CNAB invocation image... The push refers to repository [my-registry.comt/develop/prakash/nginx] 2a805e039dc0: Preparing 695758e121a1: Preparing ce59be84e3a4: Preparing 604a64d9f0ed: Preparing 75403ef7a264: Preparing b18299306266: Preparing f01953646cff: Preparing 39a89d2e4f5a: Preparing 78bbff9102e7: Preparing 63ea75ccbf0c: Preparing b18299306266: Waiting f01953646cff: Waiting 39a89d2e4f5a: Waiting 78bbff9102e7: Waiting 63ea75ccbf0c: Waiting ce59be84e3a4: Layer already exists 604a64d9f0ed: Layer already exists 2a805e039dc0: Layer already exists 695758e121a1: Layer already exists 75403ef7a264: Layer already exists b18299306266: Layer already exists f01953646cff: Layer already exists 39a89d2e4f5a: Layer already exists 78bbff9102e7: Layer already exists 63ea75ccbf0c: Layer already exists ec246d2decab3db00cea9b1053a41052: digest: sha256:55b76fa1802b0029f4c597303861869e1ffac925d096b7ebc2f6ae1f21102e79 size: 2424

Rewriting CNAB bundle.json... DEBUG name: exec DEBUG pkgDir: /Users/prakash/.porter/mixins/exec DEBUG file:
DEBUG stdin:

/Users/prakash/.porter/mixins/exec/exec version --output json --debug DEBUG name: kubernetes DEBUG pkgDir: /Users/prakash/.porter/mixins/kubernetes DEBUG file:
DEBUG stdin:

/Users/prakash/.porter/mixins/kubernetes/kubernetes version --output json --debug Starting to copy image my-registry.com/develop/prakash/nginx:ec246d2decab3db00cea9b1053a41052... Completed image my-registry.com/develop/prakash/nginx:ec246d2decab3db00cea9b1053a41052 copy Starting to copy image my-registry.com/develop/prakash/assets-nginx@sha256:9123250178a23334761946ac478aa56ec28f3f6402a491bb154fa13c724482b9... Failed to copy image my-registry.com/develop/prakash/assets-nginx@sha256:9123250178a23334761946ac478aa56ec28f3f6402a491bb154fa13c724482b9: failed to authorize: failed to fetch oauth token: unexpected status: 500 Internal Server Error error preparing the bundle with cnab-to-oci before pushing: failed to authorize: failed to fetch oauth token: unexpected status: 500 Internal Server Error Error: error preparing the bundle with cnab-to-oci before pushing: failed to authorize: failed to fetch oauth token: unexpected status: 500 Internal Server Error error preparing the bundle with cnab-to-oci before pushing: failed to authorize: failed to fetch oauth token: unexpected status: 500 Internal Server Error Closing plugins

Is --insecure-registrsy flag not being passed to subsequent images section?

But if we use --reference flag to porter publish, it works. like below

orter publish --registry my-registry.com/develop/prakash/ --reference my-registry.com/
develop/prakash/assets-nginx:0.0.2 --insecure-registry

But we thinking to publish the main app bundle. Idea is, before we install the app bundle, our backend will do porter inspect and extract the assets images. We thought adding images section to porter.yaml would help us. Please let us know the pointers to above error messages.

Where have you looked already to figure this out? didn't find relevant docs

Was the existing documentation unclear or had gaps? N/A

[carolynvs]

I am guessing that the images you are referencing are not public? You can authenticate to a private registry by running docker login my-registry.com first and then porter publish should work.

[prakashmirji]

our registry repo is private but authenticated with username/password. docker login my-registry.com succeeds. But I still get the same error when I run porter publish. BTW - as mentioned above, porter publish works fine if I pass --reference flag with value as assets image.

[carolynvs]

When you specify --registry, it overrides the calculated destination where the bundle is pushed. When you set --reference porter doesn't even try to calculate the destination. After that the commands are exactly the same, so it appears to be a problem with your authorization to the calculated destination.

Real quick, which version of Porter are you using? Older versions push to different destinations than the new versions of Porter.

[prakashmirji]

Thank you for the response. I tried with latest v1.0.0-alpha.19, I get the same error.

[prakashmirji]

If we remove the images section from porter.yaml, porter publish with same registry works. Looks like --insecure-registry is not being passed to the next set of calls.

For example, in the error messages below, main app image nginx pushed but the next image is assets-nginx (comes from image section) is failed to copy.

/Users/prakash/.porter/mixins/kubernetes/kubernetes version --output json --debug
Starting to copy image my-registry.com/develop/prakash/nginx:ec246d2decab3db00cea9b1053a41052@sha256:3dc4f3d9b14f21aee3b97e82687e118063c6f20a2fdc9a28a5b91925e87a6986...
Completed image my-registry.com/develop/prakash/nginx:ec246d2decab3db00cea9b1053a41052@sha256:3dc4f3d9b14f21aee3b97e82687e118063c6f20a2fdc9a28a5b91925e87a6986 copy
Starting to copy image my-registry.com/develop/prakash/assets-nginx@sha256:9123250178a23334761946ac478aa56ec28f3f6402a491bb154fa13c724482b9...
Failed to copy image my-registry.com/develop/prakash/assets-nginx@sha256:9123250178a23334761946ac478aa56ec28f3f6402a491bb154fa13c724482b9: failed to authorize: failed to fetch oauth token: unexpected status: 500 Internal Server Error
error preparing the bundle with cnab-to-oci before pushing: failed to authorize: failed to fetch oauth token: unexpected status: 500 Internal Server Error
error preparing the bundle with cnab-to-oci before pushing: failed to authorize: failed to fetch oauth token: unexpected status: 500 Internal Server Error
Closing plugins

[carolynvs]

Thanks for identifying the problem! I'll flag this as a bug and add it to the v1 milestone.

[prakashmirji]

Thank you. In the meantime, can you suggest any workaround?

[carolynvs]

Not really sorry. Someone just needs to look into how we pass insecure-registry to our cnab-to-oci calls and fix how it's passed.

[prakashmirji]

okay - we were debugging cnab-to-oci library code, yet to figure out the fix

[carolynvs]

I think I have the problem reproducing? It's not the same error, yours is oauth, mine is about certs signed by unknown authorities. I checked how we are calling cnab-to-oci and we are setting the registry as insecure so I'm not 100% sure what's causing the problem.

I'm not sure why you get an oauth error, are you using docker registry or is it behind more layers of middleware?

[carolynvs]

I guess I am assuming that your registry is using a self-signed certificate, is that the case?

[prakashmirji]

we use harbor registry with credentials and self-signed certs. We do docker login before push and we pass --insecure-registry flag. In your case, is your registry have credentials?

[carolynvs]

Yeah I configured the registry with a self signed cert (I'm using docker registry directly). Did you configure your docker daemon to trust the self signed certs used by the registry?

I'm not sure that what I am doing locally is reproducing what you are seeing. Can you look in the logs about that 500 error? That doesn't seem like a problem with ignoring the self-signed certs, if the request made it to the server and then the server errored out.

[kcsraju]

First of all, appreciate the help @carolynvs.

if it were an issue on local host we shouldn't be able to push any images, right?

This works:

Starting to copy image my-registry.com/develop/prakash/nginx:ec246d2decab3db00cea9b1053a41052@sha256:3dc4f3d9b14f21aee3b97e82687e118063c6f20a2fdc9a28a5b91925e87a6986...
Completed image my-registry.com/develop/prakash/nginx:ec246d2decab3db00cea9b1053a41052@sha256:3dc4f3d9b14f21aee3b97e82687e118063c6f20a2fdc9a28a5b91925e87a6986 copy

This does not:

Starting to copy image my-registry.com/develop/prakash/assets-nginx@sha256:9123250178a23334761946ac478aa56ec28f3f6402a491bb154fa13c724482b9...
Failed to copy image my-registry.com/develop/prakash/assets-nginx@sha256:9123250178a23334761946ac478aa56ec28f3f6402a491bb154fa13c724482b9: failed to authorize: failed to fetch oauth token: unexpected status: 500 Internal Server Error

Both those steps are executed in the same build context. The only diff between the two images is that the second one is listed under the "images" section in porter.yaml.

[prakashmirji]

okay - if your docker registry is not configured with credentials, can you please set credentials and give it a try.

In our case, it appears that credentials are not honored when copying 2nd image.

[carolynvs]

ah okay the referenced image I was testing with wasn't on a private registry. Trying that out now

[carolynvs]

So I pulled the carolynvs/whalesayd image from docker and pushed it to my private registry (which is using credentials, via htpasswd, not oauth). Then I updated the examples/airgap bundle to reference the image in the private registry. When I publish the bundle to the same private registry, it isn't reproducing for me:

insecure registries:  [192.168.0.110:443]
Starting to copy image 192.168.0.110:443/whalegap:e81042a6fe198ef44b24dc956b3ea974@sha256:b1a7135e3bcc64a052f82c1f264bf326b0e4f12d40a7008bbcf4823a2e9d55d8...
Completed image 192.168.0.110:443/whalegap:e81042a6fe198ef44b24dc956b3ea974@sha256:b1a7135e3bcc64a052f82c1f264bf326b0e4f12d40a7008bbcf4823a2e9d55d8 copy
Starting to copy image 192.168.0.110:443/carolynvs/whalesayd@sha256:499f71eec2e3bd78f26c268bbf5b2a65f73b96216fac4a89b86b5ebf115527b6...
Completed image 192.168.0.110:443/carolynvs/whalesayd@sha256:499f71eec2e3bd78f26c268bbf5b2a65f73b96216fac4a89b86b5ebf115527b6 copy
Bundle tag 192.168.0.110:443/whalegap:v0.2.0 pushed successfully, with digest "sha256:2f9c1525693338d16bfa4d590f8e530f8ba8ad00ebf1f7c0df18be2070e16460"

Are you able to pull that image that isn't working using the same creds that you are using with porter? docker pull my-registry.com/develop/prakash/assets-nginx@sha256:9123250178a23334761946ac478aa56ec28f3f6402a491bb154fa13c724482b9

[prakashmirji]

okay - thank you for trying that. Here is the docker put output

docker pull bd-harbor-registry.xxxx.storage.xxxx.net/develop/prakash/assets-nginx@sha256:9123250178a23334761946ac478aa56ec28f3f6402a491bb154fa13c724482b9
bd-harbor-registry.xxxx.storage.xxxx.net/develop/prakash/assets-nginx@sha256:9123250178a23334761946ac478aa56ec28f3f6402a491bb154fa13c724482b9: Pulling from develop/prakash/assets-nginx
Digest: sha256:9123250178a23334761946ac478aa56ec28f3f6402a491bb154fa13c724482b9
Status: Image is up to date for bd-harbor-registry.xxxx.storage.xxxx.net/develop/prakash/assets-nginx@sha256:9123250178a23334761946ac478aa56ec28f3f6402a491bb154fa13c724482b9
bd-harbor-registry.xxxx.storage.xxxx.net/develop/prakash/assets-nginx@sha256:9123250178a23334761946ac478aa56ec28f3f6402a491bb154fa13c724482b9

[carolynvs]

Sorry I'm stumped, with without being able to reproduce locally I'm not sure how to help debug this.

I have a branch of cnab-to-oci that prints out additional error messages that may help narrow down where in fixupBundle the call is failing if you want to try to debug locally https://github.com/cnabio/cnab-to-oci/pull/113

[prakashmirji]

Thank you.. Can you please share your porter.yaml and bundle.json you used for reproducing.

To debug with this branch https://github.com/cnabio/cnab-to-oci/pull/113, I can run below command right?

./bin/cnab-to-oci push bundle.json --target bd-harbor-registry.xxxx.storage.xxxx.net/develop/prakash --insecure-registries "bd-harbor-registry.xxxx.storage.xxxx.net/develop/prakash" --log-level debug

[prakashmirji]

I tried to test with your branch.

~/go-workspace/src/github.com/carolyn/cnab-to-oci/examples/helloworld-cnab (master*) » ../../bin/cnab-to-oci push bundle.json --target bd-harbor-registry.mip.storage.hpecorp.net/develop/prakash --insecure-registries "bd-harbor-registry.mip.storage.hpecorp.net/develop/prakash" --log-level debug

DEBU[0001] Fixing up bundle bd-harbor-registry.mip.storage.xxxx.net/develop/prakash DEBU[0001] Updating entry in relocation map for "bd-harbor-registry.mip.storage.xxxx.net/develop/prakash/helloworld:0.1.1" Starting to copy image bd-harbor-registry.mip.storage.xxxx.net/develop/prakash/helloworld:0.1.1... The push refers to repository [bd-harbor-registry.mip.storage.xxxx.net/develop/prakash] b2b5c54a04ff: Preparing 1c7a26c6ffb4: Preparing 73046094a9b8: Preparing b2b5c54a04ff: Mounted from develop/prakash/helloworld 73046094a9b8: Mounted from develop/prakash/helloworld 1c7a26c6ffb4: Mounted from develop/prakash/helloworld latest: digest: sha256:a59a4e74d9cc89e4e75dfb2cc7ea5c108e4236ba6231b53081a9e2506d1197b6 size: 942 Failed to copy image bd-harbor-registry.mip.storage.xxxx.net/develop/prakash/helloworld:0.1.1: failed to resolve or push image for service "InvocationImage" Error: failed to resolve or push image for service "InvocationImage"

~/go-workspace/src/github.com/carolyn/cnab-to-oci/examples/helloworld-cnab (master*) » docker pull bd-harbor-registry.mip.storage.xxxx.net/develop/prakash/helloworld:0.1.1

0.1.1: Pulling from develop/prakash/helloworld Digest: sha256:a59a4e74d9cc89e4e75dfb2cc7ea5c108e4236ba6231b53081a9e2506d1197b6 Status: Image is up to date for bd-harbor-registry.mip.storage.xxxx.net/develop/prakash/helloworld:0.1.1 bd-harbor-registry.mip.storage.xxxx.net/develop/prakash/helloworld:0.1.1

Here is the bundle.json

{
  "schemaVersion": "v1.0.0",
  "name": "helloworld",
  "version": "0.1.1",
  "description": "A short description of your bundle",
  "keywords": [
    "helloworld",
    "cnab",
    "tutorial"
  ],
  "maintainers": [
    {
      "name": "Jane Doe",
      "email": "jane.doe@example.com",
      "url": "https://example.com"
    }
  ],
  "invocationImages": [
    {
      "imageType": "docker",
      "image": "bd-harbor-registry.mip.storage.xxxx.net/develop/prakash/helloworld:0.1.1",
      "size": 42
    }
  ],
  "images": null,
  "parameters": null,
  "credentials": null
}

[prakashmirji]

cnab-to-oci push worked after adding digest and size params to the InvocationImages{}.

[prakashmirji]

quick update: In the images section ( of porter.yaml), tried to use my private docker repo instead of our internal harbor registry, this time porter publish command worked. Looks like remotes.CreateResolver() is not returning correct remotes.Resolver for the internal registry.

[carolynvs]

Is there anything unique about the internal harbor registry? Is it a vanilla installation of harbor 2 that uses oauth?

[jemgoss]

A few of my observations for this issue:

I don't believe this is a problem with propagation of the --insecure-registry flag for referenced images. We've not had to use that flag when publishing to our harbor registry since porter/cnab-to-oci was fixed a while back.

This only seems to happen when referencing an image from the same harbor registry. There seems to be a problem when uploading (POSTing) the blob to the target. But it's inconsistent. I had one bundle that would successfully publish and another bundle that referenced the same image that failed consistently. After deleting and retrying the successfully pushed target, it appeared the blobs still existed in the deleted target repo (a GET returned 200 despite the target clearly being deleted), so there was no attempt to recopy those blobs and the push succeeded again. I think that's down to harbor's caching. Unfortunately, I don't have admin access to harbor to force a garbage collection, which is done nightly, but I'll do some more digging today now that the cache has been cleared.

[jemgoss]

@carolynvs in terms of the error received, using some of my own logging in cnab-to-oci, it fails when it tries to copy a blob from the referenced image (bd-harbor-registry.mip.storage.hpecorp.net/jem/alpine) to the target repo (https://bd-harbor-registry.mip.storage.hpecorp.net/v2/jem/publishrepo/pywebserver/).

DEBU[0005] checking and pushing to                       url="https://bd-harbor-registry.mip.storage.hpecorp.net/v2/jem/publishrepo/pywebserver/blobs/sha256:d4ff818577bc193b309b355b02ebc9220427090057b54a59e73b79bdfe139b83"
DEBU[0005] do request                                    request.header.accept="application/vnd.docker.container.image.v1+json, */*" request.header.user-agent=containerd/1.6.1+unknown request.method=HEAD url="https://bd-harbor-registry.mip.storage.hpecorp.net/v2/jem/publishrepo/pywebserver/blobs/sha256:d4ff818577bc193b309b355b02ebc9220427090057b54a59e73b79bdfe139b83"
DEBU[0006] fetch response received                       response.header.connection=keep-alive response.header.content-length=157 response.header.content-type="application/json; charset=utf-8" response.header.date="Mon, 18 Apr 2022 17:31:48 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server=nginx response.header.set-cookie="sid=420c30c4d22a1126743a3ca5948f9b9e; Path=/; HttpOnly" response.header.x-request-id=b4011f3e-8a7e-4313-95d9-4bdeadf81ecc response.status="404 Not Found" url="https://bd-harbor-registry.mip.storage.hpecorp.net/v2/jem/publishrepo/pywebserver/blobs/sha256:d4ff818577bc193b309b355b02ebc9220427090057b54a59e73b79bdfe139b83"
DEBU[0006] do request                                    request.header.user-agent=containerd/1.6.1+unknown request.method=POST url="https://bd-harbor-registry.mip.storage.hpecorp.net/v2/jem/publishrepo/pywebserver/blobs/uploads/?mount=sha256:d4ff818577bc193b309b355b02ebc9220427090057b54a59e73b79bdfe139b83&from=bd-harbor-registry.mip.storage.hpecorp.net/jem/alpine"
Failed to copy image bd-harbor-registry.mip.storage.hpecorp.net/jem/alpine:latest: failed to authorize: failed to fetch oauth token: unexpected status: 500 Internal Server Error
Error: failed to authorize: failed to fetch oauth token: unexpected status: 500 Internal Server Error

[kcsraju]

Is there anything unique about the internal harbor registry? Is it a vanilla installation of harbor 2 that uses oauth?

We are running Harbor v2.1.3-b6de84c5, with a self-signed cert and local DB for authentication. It is not setup with OIDC. For all our normal/other work we have to update the local host's CA db with a root cert. From that point on docker can connect to it as a secure registry. The authorization happens as part of the normal operation when pushing and pulling images to/from the registry.

[carolynvs]

I don't have a ready harbor installation around, but when I have time (likely next week) I'll try to replicate against a similar setup.

[jemgoss]

But as I mentioned, cnab-to-oci is successfully pushing the invocation image to our harbor registry without any insecure-registry flag. And it successfully pushes referenced images to harbor. It just fails when the referenced image is also hosted on harbor. So I think that error might be misleading. Or at least it isn't an immediate auth problem. I can see through the logs that the other uploads are correctly using my docker config for auth and it validates the CA using my host's trust store.

[jemgoss]

A couple of updates:

1. This might be our problem: https://github.com/goharbor/harbor/issues/10305 If that is the root cause, it looks to have been fixed in Harbor v2.2.1.

2. After more experimentation, the push error occurs in pushWithAnnotation() in the cnab-to-oci mount.go code. If I disable the annotation that persuades containerd to perform a mount instead of a push, then our pushes always work.

   func pushWithAnnotation(ctx context.Context, pusher remotes.Pusher, ref reference.Named, desc ocischemav1.Descriptor) (content.Writer, error) {
   // Add the distribution source annotation to help containerd
   // mount instead of push when possible.
   // repo := fmt.Sprintf("%s.%s", labelDistributionSource, reference.Domain(ref))
   // desc.Annotations = map[string]string{
   //  repo: reference.FamiliarName(ref),
   // }
   return pusher.Push(ctx, desc)

   I'm hoping this is just a workaround to the issue above.

[carolynvs]

What is the cnab-to-oci command that you are running (which works)? I can compare that callpath to what we are doing in porter.

[jemgoss]

Steps to reproduce:

Starting with sample bundle, verify it can be pushed to harbor:

$ curl -OL https://github.com/cnabio/cnab-to-oci/blob/main/examples/helloworld-cnab/bundle.json
$ cnab-to-oci push bundle.json --target harborregistry/project/hello:v1.0.0 --auto-update-bundle
Starting to copy image cnab/helloworld:0.1.1...
Completed image cnab/helloworld:0.1.1 copy
Pushed successfully, with digest "sha256:55c6da48fe8553696bf3e2c940ea667e2d2fda3d55614a19dbd7d084780c0807"

Update the images section to include an external image and observe it copies that image as well:

  "images": {
    "alpine":{
      "description": "An included image from docker",
      "image": "alpine:latest",
      "imageType":"docker"
    }
  },

$ cnab-to-oci push bundle-docker-img.json --target harborregistry/project/hello:v1.0.0 --auto-update-bundle
Starting to copy image cnab/helloworld:0.1.1...
Completed image cnab/helloworld:0.1.1 copy
Starting to copy image alpine:latest...
Completed image alpine:latest copy
Pushed successfully, with digest "sha256:7d365cf7873e9cf403c0ff95a31a9581ebf5b2bbfea3ffc110a0f98e40f04041"

Push the alpine image to harbor, reference that image and observe failure:

$ docker tag alpine:latest harborregistry/project/alpine:latest
$ docker push harborregistry/project/alpine:latest

  "images": {
    "alpine":{
      "description": "An included image from harbor",
      "image": "harborregistry/project/alpine:latest",
      "imageType":"docker"
    }
  },

$ cnab-to-oci push bundle-harbor-img.json --target harborregistry/project/hello:v1.0.0 --auto-update-bundle
Starting to copy image cnab/helloworld:0.1.1...
Completed image cnab/helloworld:0.1.1 copy
Starting to copy image harborregistry/project/alpine:latest...
Failed to copy image harborregistry/project/alpine:latest: failed to authorize: failed to fetch oauth token: unexpected status: 500 Internal Server Error
Error: failed to authorize: failed to fetch oauth token: unexpected status: 500 Internal Server Error

[jemgoss]

I can confirm that when using a later version of Harbor (Version v2.5.0) this problem goes away. Therefore this issue can be closed.

[carolynvs]

That's great news! Thanks for letting us know.