Repositories/images not visible in UI : blob unknown to registry

[ixxeL2097]

I have installed harbor registry with the helm chart in my kubernetes cluster. Everything seems to work, I am able to login to the web UI, and also login to the registry (using skopeo) but I can't see any of the images uploaded in the web UI. My values.yaml use following settings :

• expose.type = clusterIP
• expose.tls.enabled = false
• expose.clusterIP.name = harbor
• externalURL = http://harbor

I access through the web UI using the harbor service pointing to the nginx container :

[root@workstation ~ ]$ k get svc                                                                                                                                                                                                            Thu Dec 23 12:39:14 2021
NAME                          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
harbor                        ClusterIP   10.43.198.120   <none>        80/TCP              26m
harbor-registry-chartmuseum   ClusterIP   10.43.22.8      <none>        80/TCP              26m
harbor-registry-core          ClusterIP   10.43.228.79    <none>        80/TCP              26m
harbor-registry-database      ClusterIP   10.43.119.252   <none>        5432/TCP            26m
harbor-registry-jobservice    ClusterIP   10.43.148.145   <none>        80/TCP              26m
harbor-registry-portal        ClusterIP   10.43.125.113   <none>        80/TCP              26m
harbor-registry-redis         ClusterIP   10.43.31.154    <none>        6379/TCP            26m
harbor-registry-registry      ClusterIP   10.43.98.189    <none>        5000/TCP,8080/TCP   26m
harbor-registry-trivy         ClusterIP   10.43.232.28    <none>        8080/TCP            26m

[root@workstation ~ ]$ k get pods                                                                                                                                                                                                           Thu Dec 23 12:45:08 2021
NAME                                           READY   STATUS    RESTARTS   AGE
harbor-registry-chartmuseum-7bd595b954-cdkgn   1/1     Running   0          15m
harbor-registry-core-6559c9b6bc-l4fwm          1/1     Running   0          15m
harbor-registry-database-0                     1/1     Running   0          27m
harbor-registry-jobservice-7b6dfb469c-zhw5w    1/1     Running   1          15m
harbor-registry-nginx-66697d7d55-6ctrj         1/1     Running   0          27m
harbor-registry-portal-6fdf544577-lbt5s        1/1     Running   0          27m
harbor-registry-redis-0                        1/1     Running   0          27m
harbor-registry-registry-5cf656c958-xwb7n      2/2     Running   0          15m
harbor-registry-trivy-0                        1/1     Running   0          27m

I can access the Web UI simply with a port-forward for the moment, and also expose my registry with same technique :

k port-forward harbor-registry-nginx-66697d7d55-6ctrj 8080:8080
k port-forward harbor-registry-registry-5cf656c958-xwb7n 5000:5000

and transfer images using skopeo :

[root@workstation ~ ]$ sudo skopeo login --tls-verify=false localhost:5000                                                                                                                                                              
Username: admin
Password:
Login Succeeded!

[root@workstation ~ ]$ sudo skopeo copy --insecure-policy docker://alpine:latest docker://localhost:5000/library/alpine:latest --dest-tls-verify=false 
Getting image source signatures
Copying blob 59bf1c3509f3 [--------------------------------------] 0.0b / 0.0b
Copying config c059bfaa84 [======================================] 1.4KiB / 1.4KiB
Writing manifest to image destination
Storing signatures

I also tried transfering from an inside cluster container and same result. For unknow reason, I cannot see the uploaded repositories in my Harbor web UI. If I do curl the registry, I can see that the images are properly uploaded :

[root@workstation ~ ]$ curl -k --user admin:admin localhost:5000/v2/_catalog                                                                                                                                                                Thu Dec 23 12:52:10 2021
{"repositories":["library/alpine","library/mongodb","library/nginx"]}

I tried many things, like downgrading the Harbor version but nothing worked. I am kinda stuck at the moment and cannot use Harbor. Any idea here ?

I noticed these errors in the registry pod:

time="2021-12-23T14:44:56.949749839Z" level=error msg="response completed with error" auth.user.name=admin err.code="blob unknown" err.detail=sha256:7b1a6ab2e44dbac178598dabe7cff59bd67233dba0b27e4fbd1f9d4b3c877a54 err.message="blob unknown to registry" go.version=go1.15.12 http.request.host="harbor-registry-registry.harbor.svc.cluster.local:5000" http.request.id=2f729026-0b67-4a15-a920-1b790f85f5b6 http.request.method=HEAD http.request.remoteaddr="10.42.1.101:50720" http.request.uri="/v2/library/ubuntu/blobs/sha256:7b1a6ab2e44dbac178598dabe7cff59bd67233dba0b27e4fbd1f9d4b3c877a54" http.request.useragent="skopeo/1.5.2" http.response.contenttype="application/json; charset=utf-8" http.response.duration=97.996033ms http.response.status=404 http.response.written=157 vars.digest="sha256:7b1a6ab2e44dbac178598dabe7cff59bd67233dba0b27e4fbd1f9d4b3c877a54" vars.name="library/ubuntu"
time="2021-12-23T14:44:57.820631077Z" level=error msg="response completed with error" auth.user.name=admin err.code="blob unknown" err.detail=sha256:ba6acccedd2923aee4c2acc6a23780b14ed4b8a5fa4e14e252a23b846df9b6c1 err.message="blob unknown to registry" go.version=go1.15.12 http.request.host="harbor-registry-registry.harbor.svc.cluster.local:5000" http.request.id=9af917a9-e27a-4893-adda-2b454ad47988 http.request.method=HEAD http.request.remoteaddr="10.42.1.101:50762" http.request.uri="/v2/library/ubuntu/blobs/sha256:ba6acccedd2923aee4c2acc6a23780b14ed4b8a5fa4e14e252a23b846df9b6c1" http.request.useragent="skopeo/1.5.2" http.response.contenttype="application/json; charset=utf-8" http.response.duration=85.941448ms http.response.status=404 http.response.written=157 vars.digest="sha256:ba6acccedd2923aee4c2acc6a23780b14ed4b8a5fa4e14e252a23b846df9b6c1" vars.name="library/ubuntu"

I also tried to set : registry.relativeurls = true as mentionned in the post #174 but still, same error of blob unknow to registry

[zyyw]

Hi @ixxeL2097 , are you sure the blobs are successfully copied over to harbor via skopeo? Because I noticed the following message from the description you provided:

Copying blob 59bf1c3509f3 [--------------------------------------] 0.0b / 0.0b

when the size of blobs being copied over is 0.0b.

[zyyw]

you should probable login through externalURL and using skopeo copy images over to externalURL of harbor instead of registry.

[ixxeL2097]

Yes, the image is properly copied. I can even download it from the registry after uploading it. The reason why it is 0.0b is because I did a second copy of the same image so it didn't upload it a second time, but you can notice on my curl -k --user admin:admin localhost:5000/v2/_catalog command that the images are indeed present inside the registry.

The externalURL cannot be reached from outside the cluster since it's a clusterIP service and according to annotation :

# Format: protocol://domain[:port]. Usually:
# 1) if "expose.type" is "ingress", the "domain" should be
# the value of "expose.ingress.hosts.core"
# 2) if "expose.type" is "clusterIP", the "domain" should be
# the value of "expose.clusterIP.name"
# 3) if "expose.type" is "nodePort", the "domain" should be
# the IP address of k8s node

So the externalURL in my case is just the name of the clusterIP service, not my DNS hostname. In order to make my clusterIP service reachable from outside of my cluster, I created an ingress (actually it's a virtualService since I am using istio but it's the same mechanic).

Moreover, if I already tried to login to the actual ingress linking my Harbor service (not the registry), but login is denied. The only way to login into the registry is to login against the actual harbor registy URL (weither it is inside the cluster or outside).

[LosaMiguel]

Has there been any updates on this? Many of the setting mentioned above are the same except I'm running it on OCP so I have routes (ingresses) in front of the services to reach them.

The only way to reach the registry is by routing the traffic directly to the registry pod, but the same issues arise if I do it this way.

I'm having the same problem as well, projects removed by the UI aren't updated in the registry itself and uploads to the registry aren't visible in the UI.

[hajnalmt]

For us, this same error is coming when we try to replicate an entire project from one harbor to another.

[counter2015]

I also meet similary problem.

After migrate habror to new instantce, some part of old helm charts disappeared.

$ helm search repo myrepo
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /Users/counter/.kube/config
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /Users/counter/.kube/config
NAME                    CHART VERSION   APP VERSION DESCRIPTION
newrepo/myrepo  0.1.13          1.0.0       myrepo all application charts
oldrepo/myrepo          0.1.13          1.0.0       myrepo all application charts
oldrepo/myrepo-jobs 0.1.1           1.16.0      dump and restore myrepo data

I try to reupload myrepo-jobs-0.1.1.tgz from harbor UI. It told me it's uploaded successfully, however, I cant found the chart.

My workaroud is package a new version myrepo-jobs-0.1.2.tgz of chart and reupload it.

Now it can be found from harbor UI, but the old version missed.

[github-actions[bot]]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.