search

goharbor / harbor

An open source trusted cloud native registry project that stores, signs, and scans content.
https://goharbor.io
Apache License 2.0
24.09k stars 4.76k forks source link

Harbor Proxy should serve manifests from local repository if the remote manifest digest matches a local manifest #21122

Open raphaelzoellner opened 6 days ago

raphaelzoellner commented 6 days ago

Is your feature request related to a problem? Please describe. DockerHub Proxy GET /manifest requests referencing a tag are not served from cache even when the remote manifest matches a manifest in the local repository.

This leads to requests reducing the users remainder of the rate limit.

This can be observed by running harbor with log level debug and fetching a manifest multiple times.

GET https://harbor.example.com/v2/docker-hub/goharbor/harbor-exporter/manifests/v2.10.2

[DEBUG] [/controller/proxy/controller.go:196]: Digest is not found in manifest list cache, key=cache:manifestlist:docker-hub/goharbor/harbor-exporter:sha256:b6bb051a967de0948992f4a44ed00369adcc04ba88cdc76ed8ddeb4326ccf8be
[DEBUG] [/server/middleware/repoproxy/proxy.go:203]: the tag is v2.10.2, digest is 
[WARNING] [/server/middleware/repoproxy/proxy.go:207]: Artifact: docker-hub/goharbor/harbor-exporter:v2.10.2, digest: is not found in proxy cache, fetch it from remote repo

https://github.com/goharbor/harbor/blob/v2.10.2/src/controller/proxy/controller.go#L193-L201C1 https://github.com/goharbor/harbor/blob/v2.10.2/src/server/middleware/repoproxy/proxy.go#L203-L209

Describe the solution you'd like Harbor should serve the manifest from the local repository if the remote manifest digest matches the digest of the manifest in the local repository.

Describe the main design/architecture of your solution Since Harbor already attempts to use a cache for ManifestLists I suggest to extend this behavior to Manifests by attempting to pull the manifest from the local repository.

https://github.com/goharbor/harbor/blob/v2.10.2/src/controller/proxy/controller.go#L159

stonezdj commented 1 day ago

Could you please double check the artifact with the same tag and same digest exist in the proxy cache project? It seems that the artifact didn't cached yet from the log.

There are two caches, one is the registry, another is the redis cache, the following log is refer the redis cache. [/controller/proxy/controller.go:196]: Digest is not found in manifest list cache, key=cache:manifestlist:docker-hub/goharbor/harbor-exporter:sha256:b6bb051a967de0948992f4a44ed00369adcc04ba88cdc76ed8ddeb4326ccf8be

raphaelzoellner commented 1 day ago

Sure, I've reproduced the behavior.

I've performed multiple GET manifest requests referencing the tag. https://harbor.example.com/v2/docker-hub/goharbor/harbor-exporter/manifests/v2.10.2

2024-11-05T09:26:28Z [WARNING] [/server/middleware/repoproxy/proxy.go:207]: Artifact: docker-hub/goharbor/harbor-exporter:v2.10.2, digest: is not found in proxy cache, fetch it from remote repo
2024-11-05T09:26:52Z [WARNING] [/server/middleware/repoproxy/proxy.go:207]: Artifact: docker-hub/goharbor/harbor-exporter:v2.10.2, digest: is not found in proxy cache, fetch it from remote repo
2024-11-05T09:27:47Z [WARNING] [/server/middleware/repoproxy/proxy.go:207]: Artifact: docker-hub/goharbor/harbor-exporter:v2.10.2, digest: is not found in proxy cache, fetch it from remote repo
2024-11-05T09:31:22Z [WARNING] [/server/middleware/repoproxy/proxy.go:207]: Artifact: docker-hub/goharbor/harbor-exporter:v2.10.2, digest: is not found in proxy cache, fetch it from remote repo

The corresponding artifact in the Harbor portal does not seem to have a tag associated with it, but was already created before the last GET request. grafik

On the other hand when performing a GET manifest request referencing the digest, logs of the form is not found in proxy cache, fetch it from remote repo seem not to be created, I assume the manifest is served from the local repository in this case. https://harbor.example.com/v2/docker-hub/goharbor/harbor-exporter/manifests/sha256:b6bb051a967de0948992f4a44ed00369adcc04ba88cdc76ed8ddeb4326ccf8be