Issue: Docker Images Not Displaying in Harbor UI with S3 Storage (Helm Charts are visible)

[ujala-singh]

I have deployed harbor version v2.11.0 in kubernetes. I am using S3 for imageChartStorage with the below config:

imageChartStorage:
    type: s3
    s3:
      region: us-east-1
      bucket: mybucket
      accesskey: ************
      secretkey: ************
      rootdirectory: /harbor/test/

With the above config I am able to push the docker images and helm charts successfully in my bucket under the path /harbor/test/. But on the UI under project I am only able to see the helm chart not the images. Below errors I am getting in registry pods:

harbor-test-registry-796c7c5df9-2kjtm registry time="2024-07-31T11:28:18.334962549Z" level=error msg="response completed with error" auth.user.name="harbor_registry_user" err.code="blob unknown" err.detail=sha256:f99ce65d3251b7e18b43b4f0c58a56e83c94482c41d0716a3c7d1b5a37193981 err.message="blob unknown to registry" go.version=go1.22.3 http.request.host=harbor-test.atlan.dev http.request.id=edfa243c-5212-4104-8835-af14c9524913 http.request.method=HEAD http.request.remoteaddr=10.188.25.92 http.request.uri="/v2/test_project/kube-burner/blobs/sha256:f99ce65d3251b7e18b43b4f0c58a56e83c94482c41d0716a3c7d1b5a37193981" http.request.useragent="docker/25.0.3 go/go1.21.6 git-commit/f417435 kernel/6.6.16-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/25.0.3 \(darwin\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=71.62291ms http.response.status=404 http.response.written=157 vars.digest="sha256:f99ce65d3251b7e18b43b4f0c58a56e83c94482c41d0716a3c7d1b5a37193981" vars.name="test_project/kube-burner"
harbor-test-registry-796c7c5df9-bsrsw registry time="2024-07-31T11:28:22.037997018Z" level=error msg="response completed with error" auth.user.name="harbor_registry_user" err.code="blob unknown" err.detail=sha256:9238a32e7f2dcba2de0860643da2495f16bb78fbc98e49da8735a1ab8d0daac5 err.message="blob unknown to registry" go.version=go1.22.3 http.request.host=harbor-test.atlan.dev http.request.id=0f465562-49c9-4aa0-83a2-c472b41510bb http.request.method=HEAD http.request.remoteaddr=10.188.103.72 http.request.uri="/v2/test_project/kube-burner/blobs/sha256:9238a32e7f2dcba2de0860643da2495f16bb78fbc98e49da8735a1ab8d0daac5" http.request.useragent="docker/25.0.3 go/go1.21.6 git-commit/f417435 kernel/6.6.16-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/25.0.3 \(darwin\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=71.467588ms http.response.status=404 http.response.written=157 vars.digest="sha256:9238a32e7f2dcba2de0860643da2495f16bb78fbc98e49da8735a1ab8d0daac5" vars.name="test_project/kube-burner"
harbor-test-registry-796c7c5df9-bsrsw registry time="2024-07-31T11:28:39.886084433Z" level=error msg="response completed with error" auth.user.name="harbor_registry_user" err.code="blob unknown" err.detail=sha256:cfe4adfa6c34c5fcb6fb5957b8c22d68e66b009e8972fd65c44a2cf482c45eed err.message="blob unknown to registry" go.version=go1.22.3 http.request.host=harbor-test.atlan.dev http.request.id=8fd1380c-1297-474e-9ddd-6a1e4b1676ce http.request.method=HEAD http.request.remoteaddr=10.188.103.72 http.request.uri="/v2/test_project/kube-burner/blobs/sha256:cfe4adfa6c34c5fcb6fb5957b8c22d68e66b009e8972fd65c44a2cf482c45eed" http.request.useragent="docker/25.0.3 go/go1.21.6 git-commit/f417435 kernel/6.6.16-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/25.0.3 \(darwin\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=104.345392ms http.response.status=404 http.response.written=157 vars.digest="sha256:cfe4adfa6c34c5fcb6fb5957b8c22d68e66b009e8972fd65c44a2cf482c45eed" vars.name="test_project/kube-burner"
harbor-test-registry-796c7c5df9-2kjtm registry time="2024-07-31T11:28:45.223209897Z" level=error msg="response completed with error" auth.user.name="harbor_registry_user" err.code="manifest unknown" err.detail="unknown manifest name=test_project/kube-burner revision=sha256:2326fd0d204334a67b6582d52e426c3d79a2d8f24223794cbd933c235fd667fc" err.message="manifest unknown" go.version=go1.22.3 http.request.host="harbor-test-registry:5000" http.request.id=bece1bc7-8ee3-4849-9f16-efe380c250d3 http.request.method=GET http.request.remoteaddr="10.188.80.155:43346" http.request.uri="/v2/test_project/kube-burner/manifests/sha256:2326fd0d204334a67b6582d52e426c3d79a2d8f24223794cbd933c235fd667fc" http.request.useragent=harbor-registry-client http.response.contenttype="application/json; charset=utf-8" http.response.duration=107.778259ms http.response.status=404 http.response.written=200 vars.name="test_project/kube-burner" vars.reference="sha256:2326fd0d204334a67b6582d52e426c3d79a2d8f24223794cbd933c235fd667fc"
harbor-test-registry-796c7c5df9-2kjtm registry time="2024-07-31T11:31:48.236382956Z" level=error msg="response completed with error" auth.user.name="harbor_registry_user" err.code="manifest unknown" err.detail="unknown manifest name=test_project/kube-burner revision=sha256:2326fd0d204334a67b6582d52e426c3d79a2d8f24223794cbd933c235fd667fc" err.message="manifest unknown" go.version=go1.22.3 http.request.host="harbor-test-registry:5000" http.request.id=e4a603da-996d-47ae-b1b0-777417588846 http.request.method=GET http.request.remoteaddr="10.188.80.155:43346" http.request.uri="/v2/test_project/kube-burner/manifests/sha256:2326fd0d204334a67b6582d52e426c3d79a2d8f24223794cbd933c235fd667fc" http.request.useragent=harbor-registry-client http.response.contenttype="application/json; charset=utf-8" http.response.duration=111.131474ms http.response.status=404 http.response.written=200 vars.name="test_project/kube-burner" vars.reference="sha256:2326fd0d204334a67b6582d52e426c3d79a2d8f24223794cbd933c235fd667fc"

This is what I am getting while pushing image from CLI:

$ docker push my-harbor-domain.com/test_project/kube-burner                                                                                    
Using default tag: latest
The push refers to repository [my-harbor-domain.com/test_project/kube-burner]
cfd7ccda46d8: Pushed
ef01057d3006: Pushed
0a25e17a849f: Pushed
d1e9096d5aaa: Pushed
f9d6a68dc894: Pushed
5c29dc0d3b27: Pushed
36eed5648234: Pushed
09096b81f643: Pushed
194dafb2d78b: Pushed
a6919eaa163f: Pushed
dc9468f30913: Pushed
unknown: http status code: 404, body: {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Name":"test_project/kube-burner","Revision":"sha256:2326fd0d204334a67b6582d52e426c3d79a2d8f24223794cbd933c235fd667fc"}}]} 

This is what I am getting on S3:

This is what I am getting while pushing helm chart from CLI:

$ helm push ingress-nginx-4.11.1.tgz oci://my-harbor-domain.com/test_project  
Pushed: my-harbor-domain.com/test_project/ingress-nginx:4.11.1
Digest: sha256:008edaf9fd3bb5c74a8f73e5dfa9b4409e0c085494d0281592988257578d327

But On harbor UI I only see helm chart artifact but no image.

[stonezdj]

In Harbor, Helm charts are OCI artifacts, not docker images.

[ujala-singh]

@stonezdj I have pushed the helm charts as oci artifacts only, if you can check the very last command and output:

$ helm push ingress-nginx-4.11.1.tgz oci://my-harbor-domain.com/test_project  
Pushed: my-harbor-domain.com/test_project/ingress-nginx:4.11.1
Digest: sha256:008edaf9fd3bb5c74a8f73e5dfa9b4409e0c085494d0281592988257578d327

[ujala-singh]

I have observed one more thing that If I delete a project on harbor, its not deleting it on the S3 side.

[stonezdj]

Try to push another docker image? Delete the artifact just remove it from harbor database, it will not delete it from storage. it is the Garbage collection to remove it from storage.

[ujala-singh]

I tried to push again using below github action:

- name: Build and push docker image
  id: docker_build
  uses: docker/build-push-action@v6
  with:
    context: .
    file: ./App/Dockerfile
    push: true
    platforms: linux/amd64,linux/arm64
    tags: ${{ secrets.HARBOR_URL }}/${{ secrets.HARBOR_PROJECT }}/${{ github.event.repository.name }}-${{ steps.get_branch.outputs.branch }}:${{ steps.get_version.outputs.version }}
  env:
    DOCKER_CLIENT_TIMEOUT: 300
    COMPOSE_HTTP_TIMEOUT: 300

Now I am able to push and its visible on both the sides S3 as well as Harbor. But on harbor side I am able to see 2 more versions of the images apart from linux/amd64,linux/arm64 these two which does not have any OS Arch.

[stonezdj]

It could be an artifact accessory generated by build tools such as in-toto. the previous question might be related to the redis cache, you can run the flushdb command in the redis.

[ujala-singh]

Sometimes, my login fails from the github actions and it started happening quite frequently now:

Logging in to Harbor registry...
Error response from daemon: Get "https://harbor-test.dev/v2/": Get "https://harbor-test.dev/service/token?account=admin&client_id=docker&offline_token=true&service=harbor-registry": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) (Client.Timeout exceeded while awaiting headers)
Login failed.