Deploy --remote unable to read Harbor registry when given Administrator Credentials

[HanBrol0]

I'm trying to perform an on-cluster build for a brand new function using: kn func deploy --remote --registry=harbor-core.harbor.svc.cluster.local/repository Username=admin Password=<admin password>

The initial credential check of the registry succeeds. During the build, I receive the following error: ERROR: failed to initialize analyzer: validating registry read access: ensure registry read access to harbor-core.harbor.svc.cluster.local/project/function:latest

What I have tried so far:

1. Adding registry credentials for Tekton using the 'tekton.dev/docker-0: https://harbor-core.harbor.svc.cluster.local' annotation as well as 'tekton.dev/docker-0: https://harbor-core.harbor.svc.cluster.local/project'. Still receiving the same error.
2. Using a different administrator account. Still receiving the same error.
3. Using the external domain name for reaching the repository from outside the cluster. Still receiving the same error.
4. Creating CoreDNS rewrites to ensure that the internal cluster requests are going to the correct Kubernetes service
5. Creating an Istio VirtualService to ensure traffic is routed to the appropriate service based on the URI

Additional Info: I was looking at the Tekton Pipeline to make sure it was using the proper credentials that are provided. When looking at the Kubernetes Secret that the pipeline creates I noticed that the server does not include the project directory:

{ "auths": { "harbor-core.harbor.svc.cluster.local": { "username" : "admin", "password":"<password>", "auth":"<auth>" } } }

When using busybox to curl the harbor registry I'm able to retrieve a list of the project/repositories: curl -X GET -u "admin:<password>" http://harbor-core.harbor.svc.cluster.local/v2/_catalog

returns {"repositories":["public/function"]}

Any sort of help or additional things to look at in order to get this working would be highly appreciated.

Thanks!

Edit: I also wanted to mention that I am able to use "kn func build --push --registry=harbor.domain/project" to locally build the function and push to the repository with the Harbor admin credentials.

Edit 2: I noticed that the Tekton PipelineRuns template that is created also Creates a secret "function-pack-git-pipeline-secret" and is an Opaque type. I'm wondering if the problem might be that it is not of type "kubernetes.io/dockerconfigjson" per https://tekton.dev/docs/pipelines/auth/#basic-auth-for-docker?

[matejvasek]

@HanBrol0 is this reproducible with some other registry?

I noticed that the server does not include the project directory.

I believe this is expected: docker config has credentials per registry not per repository/namespace.

[matejvasek]

Did you use s2i or pack builder?

[matejvasek]

Do you use podman or docker?

[matejvasek]

Did you use s2i or pack builder?

I see you are using pack builder. Could you please try s2i build strategy --builder=s2i?

[HanBrol0]

@HanBrol0 is this reproducible with some other registry?

I noticed that the server does not include the project directory.

I believe this is expected: docker config has credentials per registry not per repository/namespace.

Ok, thank you for the clarification. I've been at this for a while and I think I've just started grasping at straws.

Do you use podman or docker?

When using my local builds, it's docker. The remote builds are the default templates created by the 'knative func' command which I believe is Kaniko.

Did you use s2i or pack builder?

I see you are using pack builder. Could you please try s2i build strategy --builder=s2i?

So at the moment, it's been as many defaults as possible to help improve my understanding of a lot of things. I just ran the build with the --builder=s2i command and received the following:

Getting image source signatures
Checking if image destination supports signatures
Copying blob sha256:834f52236467e6bec0d40ab7f867f4fd18f3be8087ba8caa06823c6d7f1f28f2
Copying blob sha256:8ee5f1e863ab9d2a5c9741aa4954fd8be5fa515465efdc38cb4e15808b58010e
Copying blob sha256:c2650fe947f606317fadb11c65aae44ee7b8ebd7c245a639d3521ce7a8d4544f
Copying blob sha256:50ccdb01751aeb08fdb4bb077afe0f302149e1fb82242159e5787372a750d3c1
time="2024-02-01T20:00:40Z" level=error msg="While applying layer: ApplyLayer stdout:  stderr: remount /, flags: 0x44000: permission denied exit status 1"
Error: creating build container: copying system image from manifest list: writing blob: adding layer with blob "sha256:c2650fe947f606317fadb11c65aae44ee7b8ebd7c245a639d3521ce7a8d4544f": ApplyLayer stdout:  stderr: remount /, flags: 0x44000: permission denied exit status 1

Step failed

[matejvasek]

@HanBrol0 this is weird, does docker push work all right?

[HanBrol0]

@matejvasek Yes, docker push can push to both a new and existing repository.

[matejvasek]

what if you try using pack CLI directly

pack build harbor-core.harbor.svc.cluster.local/project/function:latest -Bghcr.io/knative/builder-jammy-tiny:latest --trust-builder --publish

EDIT: you might need to use kubectl port-forward and modify your /etc/hosts to test this.

[matejvasek]

https://github.com/buildpacks/pack

[matejvasek]

And what about func local build func deploy --builder=[s2i|pack] does that work?

[HanBrol0]

@matejvasek

The following did not work: pack build harbor-core.harbor.svc.cluster.local/project/function:latest -Bghcr.io/knative/builder-jammy-tiny:latest --trust-builder --publish

However, it did work once I changed the domain to the external address, which make sense since the build is outside the cluster:
pack build harbor.domain/project/function:latest -Bghcr.io/knative/builder-jammy-tiny:latest --trust-builder --publish

Both of the following pass the analyzing stage and successfully pushes the function to the registry: func deploy --builder=[s2i|pack]

[matejvasek]

@HanBrol0 did you debug this yourself any further, any results? It's hard for me to fix this since I cannot reproduce this.

[HanBrol0]

@matejvasek I'm still working on it. Based on the last set of commands, I'm thinking it might be related to some sort of service resolution within the cluster. I'm just looking for ways to prove that at the moment.

[HanBrol0]

@matejvasek

Just wanted to give you a heads up:

Last night, I changed my infrastructure so that my Harbor registry is running on a separate machine, and not as a Kubernetes services within the cluster. After doing this, the analyze stage was able to complete successfully, though it did require retrieving a valid certificate for the deployment to succeed.

I will go back eventually to attempt to utilize Harbor within my cluster, but for now I am satisfied and look forward to building Knative functions.

Thanks for your assistance!

[github-actions[bot]]

This issue is stale because it has been open for 90 days with no activity. It will automatically close after 30 more days of inactivity. Reopen the issue with /reopen. Mark the issue as fresh by adding the comment /remove-lifecycle stale.

[github-actions[bot]]

This issue is stale because it has been open for 90 days with no activity. It will automatically close after 30 more days of inactivity. Reopen the issue with /reopen. Mark the issue as fresh by adding the comment /remove-lifecycle stale.