search

goharbor / harbor

An open source trusted cloud native registry project that stores, signs, and scans content.
https://goharbor.io
Apache License 2.0
24.29k stars 4.77k forks source link

[Feature Request] A Way to Reduce Quota Consumption for Replication on Docker Hub #20632

Open BrettDean opened 5 months ago

BrettDean commented 5 months ago

Reducing Quota Consumption for Replication on Docker Hub

Problem Description:

I’ve observed that Harbor consumes a significant amount of Docker Hub quota (free accounts have only 200 pulls every 6 hours) every time replication occurs, even though all images in this repository are already present in Harbor.

In the following example, only 4 quotas were consumed, but this is from a repository with few tags. If there were hundreds of tags, the quota consumption would be much higher.

Replication Rule:

image

worker log:

2024-06-19T23:48:15Z [INFO] [/controller/replication/transfer/image/transfer.go:139]: client for source registry [type: docker-hub, URL: https://hub.docker.com, insecure: false] created
2024-06-19T23:48:15Z [INFO] [/controller/replication/transfer/image/transfer.go:149]: client for destination registry [type: harbor, URL: http://core:8080, insecure: true] created
2024-06-19T23:48:15Z [INFO] [/controller/replication/transfer/image/transfer.go:182]: copying vergoh/vnstat:[latest,2.12,dev,2.11,2.10,2.9,2.8,2.7](source registry) to vergoh/vnstat:[latest,2.12,dev,2.11,2.10,2.9,2.8,2.7](destination registry)...
2024-06-19T23:48:15Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying vergoh/vnstat:latest(source registry) to vergoh/vnstat:latest(destination registry)...
2024-06-19T23:48:15Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact vergoh/vnstat:latest ...
2024-06-19T23:48:21Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact vergoh/vnstat:latest pulled
2024-06-19T23:48:24Z [INFO] [/controller/replication/transfer/image/transfer.go:226]: the artifact vergoh/vnstat:latest already exists on the destination registry, skip
2024-06-19T23:48:24Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying vergoh/vnstat:2.12(source registry) to vergoh/vnstat:2.12(destination registry)...
2024-06-19T23:48:24Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact vergoh/vnstat:2.12 ...
2024-06-19T23:48:24Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact vergoh/vnstat:2.12 pulled
2024-06-19T23:48:24Z [INFO] [/controller/replication/transfer/image/transfer.go:226]: the artifact vergoh/vnstat:2.12 already exists on the destination registry, skip
2024-06-19T23:48:24Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying vergoh/vnstat:dev(source registry) to vergoh/vnstat:dev(destination registry)...
2024-06-19T23:48:24Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact vergoh/vnstat:dev ...
2024-06-19T23:48:25Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact vergoh/vnstat:dev pulled
2024-06-19T23:48:26Z [INFO] [/controller/replication/transfer/image/transfer.go:226]: the artifact vergoh/vnstat:dev already exists on the destination registry, skip
2024-06-19T23:48:26Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying vergoh/vnstat:2.11(source registry) to vergoh/vnstat:2.11(destination registry)...
2024-06-19T23:48:26Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact vergoh/vnstat:2.11 ...
2024-06-19T23:48:26Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact vergoh/vnstat:2.11 pulled
2024-06-19T23:48:27Z [INFO] [/controller/replication/transfer/image/transfer.go:226]: the artifact vergoh/vnstat:2.11 already exists on the destination registry, skip
2024-06-19T23:48:27Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying vergoh/vnstat:2.10(source registry) to vergoh/vnstat:2.10(destination registry)...
2024-06-19T23:48:27Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact vergoh/vnstat:2.10 ...
2024-06-19T23:48:27Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact vergoh/vnstat:2.10 pulled
2024-06-19T23:48:28Z [INFO] [/controller/replication/transfer/image/transfer.go:226]: the artifact vergoh/vnstat:2.10 already exists on the destination registry, skip
2024-06-19T23:48:28Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying vergoh/vnstat:2.9(source registry) to vergoh/vnstat:2.9(destination registry)...
2024-06-19T23:48:28Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact vergoh/vnstat:2.9 ...
2024-06-19T23:48:28Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact vergoh/vnstat:2.9 pulled
2024-06-19T23:48:29Z [INFO] [/controller/replication/transfer/image/transfer.go:226]: the artifact vergoh/vnstat:2.9 already exists on the destination registry, skip
2024-06-19T23:48:29Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying vergoh/vnstat:2.8(source registry) to vergoh/vnstat:2.8(destination registry)...
2024-06-19T23:48:29Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact vergoh/vnstat:2.8 ...
2024-06-19T23:48:30Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact vergoh/vnstat:2.8 pulled
2024-06-19T23:48:30Z [INFO] [/controller/replication/transfer/image/transfer.go:226]: the artifact vergoh/vnstat:2.8 already exists on the destination registry, skip
2024-06-19T23:48:30Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying vergoh/vnstat:2.7(source registry) to vergoh/vnstat:2.7(destination registry)...
2024-06-19T23:48:30Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact vergoh/vnstat:2.7 ...
2024-06-19T23:48:31Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact vergoh/vnstat:2.7 pulled
2024-06-19T23:48:32Z [INFO] [/controller/replication/transfer/image/transfer.go:226]: the artifact vergoh/vnstat:2.7 already exists on the destination registry, skip
2024-06-19T23:48:32Z [INFO] [/controller/replication/transfer/image/transfer.go:204]: copy vergoh/vnstat:[latest,2.12,dev,2.11,2.10,2.9,2.8,2.7](source registry) to vergoh/vnstat:[latest,2.12,dev,2.11,2.10,2.9,2.8,2.7](destination registry) completed

Solution

After reviewing Docker Hub’s documentation regarding rate limits:

Definition of limits

A user's limit is equal to the highest entitlement of their personal account or any organization they belong to. To take advantage of this, you must sign in to Docker Hub as an authenticated user. For more information, see How do I authenticate pull requests. Unauthenticated (anonymous) users will have the limits enforced via IP.

  • Pulls are accounted to the user doing the pull, not to the owner of the image.
  • A pull request is defined as up to two GET requests on registry manifest URLs (/v2/*/manifests/*).
  • A normal image pull makes a single manifest request.
  • A pull request for a multi-arch image makes two manifest requests.
  • HEAD requests aren't counted.
  • Some images are unlimited through our Docker Sponsored Open Source and Docker Verified Publisher programs.

I've noticed that HEAD requests don't consume quotas. Harbor can compare local images with those in the official repository using HEAD requests instead of pulls, like this:

TOKEN=$(curl -s --user "username:password" "https://auth.docker.io/token?service=registry.docker.io&scope=repository:vergoh/vnstat:pull" | jq -r .token)

curl -I -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -H "Authorization: Bearer $TOKEN" https://registry.hub.docker.com/v2/vergoh/vnstat/manifests/latest
HTTP/1.1 200 OK
content-length: 3161
content-type: application/vnd.oci.image.index.v1+json
docker-content-digest: sha256:72a6f2d6c5cbc8382ad39dbc50ff3f3a23260ca282555e255742fad3bd102d06
docker-distribution-api-version: registry/2.0
etag: "sha256:72a6f2d6c5cbc8382ad39dbc50ff3f3a23260ca282555e255742fad3bd102d06"
date: Wed, 19 Jun 2024 23:45:53 GMT
strict-transport-security: max-age=31536000
ratelimit-limit: 200;w=21600
ratelimit-remaining: 192;w=21600
docker-ratelimit-source: 555a3e09-df12-4db8-a680-76282172735e

From the above request, we can obtain the docker-content-digest for vergoh/vnstat:latest: "sha256:72a6f2d6c5cbc8382ad39dbc50ff3f3a23260ca282555e255742fad3bd102d06".

This docker-content-digest can be compared with the sha256 of Harbor Artifacts to determine if the images are different, without consuming quotas.

image

This approach significantly reduces the occurrence of 429 Too Many Requests, especially when replicating hundreds of repositories.

Related:

https://github.com/goharbor/harbor/issues/19199

https://github.com/goharbor/harbor/issues/18094

MinerYang commented 5 months ago

Hi @BrettDean ,

For dockerhub adapter, it will list tags while replicating. The request would be something like GET https://my.harbor.domain/v2/repositories/<my-repository>/tags. However it not the using Get manifest which supposed to not hit the Definition of limits. Could you provide the proxy and harbor registry logs to help us triage better. Thx

BrettDean commented 5 months ago

Hi @BrettDean ,

For dockerhub adapter, it will list tags while replicating. The request would be something like GET https://my.harbor.domain/v2/repositories/<my-repository>/tags. However it not the using Get manifest which supposed to not hit the Definition of limits. Could you provide the proxy and harbor registry logs to help us triage better. Thx

Thank you for your response @MinerYang , I apologize for forgetting to provide the additional log files last time. This time, the replicate also consumed 4 quotas, decreasing from 200 to 196.

proxy.log:

Jun 24 07:51:18 172.28.0.1 proxy[2262]: 127.0.0.1 - "GET / HTTP/1.1" 308 171 "-" "curl/8.1.2" 0.000 - .
Jun 24 07:51:18 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions?policy_id=24&page_size=15&page=1 HTTP/1.1" 200 4997 "https://harbor.xxx.com/harbor/replications" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.011 0.010 .
Jun 24 07:51:32 172.28.0.1 proxy[2262]: 172.28.0.1 - "POST /api/v2.0/replication/executions HTTP/1.1" 201 0 "https://harbor.xxx.com/harbor/replications" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.047 0.047 .
Jun 24 07:51:32 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions?policy_id=24&page_size=15&page=1 HTTP/1.1" 200 4723 "https://harbor.xxx.com/harbor/replications" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.007 0.008 .
Jun 24 07:51:42 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions?policy_id=24&page_size=15&page=1 HTTP/1.1" 200 4723 "https://harbor.xxx.com/harbor/replications" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.009 0.009 .
Jun 24 07:51:43 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions/1531 HTTP/1.1" 200 224 "https://harbor.xxx.com/harbor/replications" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.004 0.004 .
Jun 24 07:51:43 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions/1531/tasks?page_size=50&page=1 HTTP/1.1" 200 316 "https://harbor.xxx.com/harbor/replications/1531/tasks" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.007 0.006 .
Jun 24 07:51:45 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions/1531/tasks/607/log HTTP/1.1" 200 2197 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.010 0.011 .
Jun 24 07:51:45 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions/1531/tasks/607/log HTTP/1.1" 200 2333 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.007 0.008 .
Jun 24 07:51:46 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions/1531/tasks/607/log HTTP/1.1" 200 2333 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.011 0.010 .
Jun 24 07:51:46 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions/1531/tasks/607/log HTTP/1.1" 200 2333 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.006 0.006 .
Jun 24 07:51:47 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions/1531/tasks/607/log HTTP/1.1" 200 2955 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.006 0.006 .
Jun 24 07:51:48 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions/1531/tasks/607/log HTTP/1.1" 200 2955 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.007 0.007 .
Jun 24 07:51:48 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions/1531/tasks/607/log HTTP/1.1" 200 2955 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.006 0.007 .
Jun 24 07:51:48 172.28.0.1 proxy[2262]: 127.0.0.1 - "GET / HTTP/1.1" 308 171 "-" "curl/8.1.2" 0.000 - .
Jun 24 07:51:52 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions?policy_id=24&page_size=15&page=1 HTTP/1.1" 200 4723 "https://harbor.xxx.com/harbor/replications/1531/tasks" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.012 0.012 .
Jun 24 07:51:53 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions/1531 HTTP/1.1" 200 224 "https://harbor.xxx.com/harbor/replications/1531/tasks" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.004 0.004 .
Jun 24 07:51:54 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions/1531/tasks?page_size=50&page=1 HTTP/1.1" 200 316 "https://harbor.xxx.com/harbor/replications/1531/tasks" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.006 0.006 .
Jun 24 07:52:03 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions?policy_id=24&page_size=15&page=1 HTTP/1.1" 200 4723 "https://harbor.xxx.com/harbor/replications/1531/tasks" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.011 0.012 .
Jun 24 07:52:04 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions/1531 HTTP/1.1" 200 224 "https://harbor.xxx.com/harbor/replications/1531/tasks" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.004 0.005 .
Jun 24 07:52:04 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions/1531/tasks?page_size=50&page=1 HTTP/1.1" 200 313 "https://harbor.xxx.com/harbor/replications/1531/tasks" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.005 0.006 .
Jun 24 07:52:09 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions/1531/tasks/607/log HTTP/1.1" 200 5857 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.008 0.009 .
Jun 24 07:52:10 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions/1531/tasks/607/log HTTP/1.1" 200 5857 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.007 0.007 .
Jun 24 07:52:13 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions?policy_id=24&page_size=15&page=1 HTTP/1.1" 200 4723 "https://harbor.xxx.com/harbor/replications/1531/tasks" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.012 0.011 .
Jun 24 07:52:14 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions/1531 HTTP/1.1" 200 224 "https://harbor.xxx.com/harbor/replications/1531/tasks" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.003 0.004 .
Jun 24 07:52:14 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions/1531/tasks?page_size=50&page=1 HTTP/1.1" 200 313 "https://harbor.xxx.com/harbor/replications/1531/tasks" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.006 0.005 .
Jun 24 07:52:18 172.28.0.1 proxy[2262]: 127.0.0.1 - "GET / HTTP/1.1" 308 171 "-" "curl/8.1.2" 0.000 - .
Jun 24 07:52:23 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions?policy_id=24&page_size=15&page=1 HTTP/1.1" 200 4720 "https://harbor.xxx.com/harbor/replications/1531/tasks" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.010 0.010 .
Jun 24 07:52:24 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions/1531 HTTP/1.1" 200 221 "https://harbor.xxx.com/harbor/replications/1531/tasks" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.004 0.005 .
Jun 24 07:52:24 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions/1531/tasks?page_size=50&page=1 HTTP/1.1" 200 313 "https://harbor.xxx.com/harbor/replications/1531/tasks" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.006 0.006 .
Jun 24 07:52:48 172.28.0.1 proxy[2262]: 127.0.0.1 - "GET / HTTP/1.1" 308 171 "-" "curl/8.1.2" 0.000 - .

registry.log:


Jun 24 07:51:07 172.28.0.1 registry[2262]: 172.28.0.8 - - [24/Jun/2024:07:51:07 +0000] "GET / HTTP/1.1" 200 0 "" "Go-http-client/1.1"
Jun 24 07:51:08 172.28.0.1 registry[2262]: 127.0.0.1 - - [24/Jun/2024:07:51:08 +0000] "GET / HTTP/1.1" 200 0 "" "curl/8.1.2"
Jun 24 07:51:17 172.28.0.1 registry[2262]: 172.28.0.8 - - [24/Jun/2024:07:51:17 +0000] "GET / HTTP/1.1" 200 0 "" "Go-http-client/1.1"
Jun 24 07:51:27 172.28.0.1 registry[2262]: 172.28.0.8 - - [24/Jun/2024:07:51:27 +0000] "GET / HTTP/1.1" 200 0 "" "Go-http-client/1.1"
Jun 24 07:51:37 172.28.0.1 registry[2262]: 172.28.0.8 - - [24/Jun/2024:07:51:37 +0000] "GET / HTTP/1.1" 200 0 "" "Go-http-client/1.1"
Jun 24 07:51:38 172.28.0.1 registry[2262]: 127.0.0.1 - - [24/Jun/2024:07:51:38 +0000] "GET / HTTP/1.1" 200 0 "" "curl/8.1.2"
Jun 24 07:51:40 172.28.0.1 registry[2262]: time="2024-06-24T07:51:40.447860968Z" level=info msg="authorized request" go.version=go1.22.3 http.request.host="core:8080" http.request.id=214da142-3a12-4e4a-ad4a-c9dfe9180fa1 http.request.method=HEAD http.request.remoteaddr=172.28.0.11 http.request.uri="/v2/vergoh/vnstat/manifests/sha256:72a6f2d6c5cbc8382ad39dbc50ff3f3a23260ca282555e255742fad3bd102d06" http.request.useragent=harbor-registry-client vars.name="vergoh/vnstat" vars.reference="sha256:72a6f2d6c5cbc8382ad39dbc50ff3f3a23260ca282555e255742fad3bd102d06" 
Jun 24 07:51:43 172.28.0.1 registry[2262]: time="2024-06-24T07:51:43.911884843Z" level=info msg="redis: connect redis:6379" go.version=go1.22.3 instance.id=9bd6e4a0-5225-4b3b-a070-c741100fd968 redis.connect.duration=6.691997ms service=registry version=v2.8.3.m 
Jun 24 07:51:44 172.28.0.1 registry[2262]: time="2024-06-24T07:51:44.726218673Z" level=info msg="response completed" go.version=go1.22.3 http.request.host="core:8080" http.request.id=214da142-3a12-4e4a-ad4a-c9dfe9180fa1 http.request.method=HEAD http.request.remoteaddr=172.28.0.11 http.request.uri="/v2/vergoh/vnstat/manifests/sha256:72a6f2d6c5cbc8382ad39dbc50ff3f3a23260ca282555e255742fad3bd102d06" http.request.useragent=harbor-registry-client http.response.contenttype="application/vnd.oci.image.index.v1+json" http.response.duration=4.285460864s http.response.status=200 http.response.written=3161 
Jun 24 07:51:44 172.28.0.1 registry[2262]: 172.28.0.8 - - [24/Jun/2024:07:51:40 +0000] "HEAD /v2/vergoh/vnstat/manifests/sha256:72a6f2d6c5cbc8382ad39dbc50ff3f3a23260ca282555e255742fad3bd102d06 HTTP/1.1" 200 3161 "" "harbor-registry-client"
Jun 24 07:51:45 172.28.0.1 registry[2262]: time="2024-06-24T07:51:45.095217224Z" level=info msg="authorized request" go.version=go1.22.3 http.request.host="core:8080" http.request.id=b26d2c34-e501-4d65-b3b3-aa0a430a0ccc http.request.method=HEAD http.request.remoteaddr=172.28.0.11 http.request.uri="/v2/vergoh/vnstat/manifests/sha256:72a6f2d6c5cbc8382ad39dbc50ff3f3a23260ca282555e255742fad3bd102d06" http.request.useragent=harbor-registry-client vars.name="vergoh/vnstat" vars.reference="sha256:72a6f2d6c5cbc8382ad39dbc50ff3f3a23260ca282555e255742fad3bd102d06" 
Jun 24 07:51:45 172.28.0.1 registry[2262]: 172.28.0.8 - - [24/Jun/2024:07:51:45 +0000] "HEAD /v2/vergoh/vnstat/manifests/sha256:72a6f2d6c5cbc8382ad39dbc50ff3f3a23260ca282555e255742fad3bd102d06 HTTP/1.1" 200 3161 "" "harbor-registry-client"
Jun 24 07:51:45 172.28.0.1 registry[2262]: time="2024-06-24T07:51:45.101121821Z" level=info msg="response completed" go.version=go1.22.3 http.request.host="core:8080" http.request.id=b26d2c34-e501-4d65-b3b3-aa0a430a0ccc http.request.method=HEAD http.request.remoteaddr=172.28.0.11 http.request.uri="/v2/vergoh/vnstat/manifests/sha256:72a6f2d6c5cbc8382ad39dbc50ff3f3a23260ca282555e255742fad3bd102d06" http.request.useragent=harbor-registry-client http.response.contenttype="application/vnd.oci.image.index.v1+json" http.response.duration=12.331753ms http.response.status=200 http.response.written=3161 
Jun 24 07:51:45 172.28.0.1 registry[2262]: time="2024-06-24T07:51:45.548040073Z" level=info msg="authorized request" go.version=go1.22.3 http.request.host="core:8080" http.request.id=7648778c-3ba1-403f-bc09-756de2e6361d http.request.method=HEAD http.request.remoteaddr=172.28.0.11 http.request.uri="/v2/vergoh/vnstat/manifests/sha256:6db175557098daba90ac4dd9ed0abfde62993d61d1d5fca949f011ca1dc233f4" http.request.useragent=harbor-registry-client vars.name="vergoh/vnstat" vars.reference="sha256:6db175557098daba90ac4dd9ed0abfde62993d61d1d5fca949f011ca1dc233f4" 
Jun 24 07:51:46 172.28.0.1 registry[2262]: time="2024-06-24T07:51:46.940810903Z" level=info msg="response completed" go.version=go1.22.3 http.request.host="core:8080" http.request.id=7648778c-3ba1-403f-bc09-756de2e6361d http.request.method=HEAD http.request.remoteaddr=172.28.0.11 http.request.uri="/v2/vergoh/vnstat/manifests/sha256:6db175557098daba90ac4dd9ed0abfde62993d61d1d5fca949f011ca1dc233f4" http.request.useragent=harbor-registry-client http.response.contenttype="application/vnd.oci.image.index.v1+json" http.response.duration=1.396110124s http.response.status=200 http.response.written=3161 
Jun 24 07:51:46 172.28.0.1 registry[2262]: 172.28.0.8 - - [24/Jun/2024:07:51:45 +0000] "HEAD /v2/vergoh/vnstat/manifests/sha256:6db175557098daba90ac4dd9ed0abfde62993d61d1d5fca949f011ca1dc233f4 HTTP/1.1" 200 3161 "" "harbor-registry-client"
Jun 24 07:51:47 172.28.0.1 registry[2262]: time="2024-06-24T07:51:47.380939307Z" level=info msg="authorized request" go.version=go1.22.3 http.request.host="core:8080" http.request.id=5913f786-692e-40a0-b139-7be49ff95f69 http.request.method=HEAD http.request.remoteaddr=172.28.0.11 http.request.uri="/v2/vergoh/vnstat/manifests/sha256:0a9a462a2d6f45af045e02c0e12b555ee369dc37340fc77abc6fab3d5489f5b3" http.request.useragent=harbor-registry-client vars.name="vergoh/vnstat" vars.reference="sha256:0a9a462a2d6f45af045e02c0e12b555ee369dc37340fc77abc6fab3d5489f5b3" 
Jun 24 07:51:47 172.28.0.1 registry[2262]: 172.28.0.8 - - [24/Jun/2024:07:51:47 +0000] "GET / HTTP/1.1" 200 0 "" "Go-http-client/1.1"
Jun 24 07:51:48 172.28.0.1 registry[2262]: 172.28.0.8 - - [24/Jun/2024:07:51:47 +0000] "HEAD /v2/vergoh/vnstat/manifests/sha256:0a9a462a2d6f45af045e02c0e12b555ee369dc37340fc77abc6fab3d5489f5b3 HTTP/1.1" 200 3161 "" "harbor-registry-client"
Jun 24 07:51:48 172.28.0.1 registry[2262]: time="2024-06-24T07:51:48.859474439Z" level=info msg="response completed" go.version=go1.22.3 http.request.host="core:8080" http.request.id=5913f786-692e-40a0-b139-7be49ff95f69 http.request.method=HEAD http.request.remoteaddr=172.28.0.11 http.request.uri="/v2/vergoh/vnstat/manifests/sha256:0a9a462a2d6f45af045e02c0e12b555ee369dc37340fc77abc6fab3d5489f5b3" http.request.useragent=harbor-registry-client http.response.contenttype="application/vnd.oci.image.index.v1+json" http.response.duration=1.482104416s http.response.status=200 http.response.written=3161 
Jun 24 07:51:49 172.28.0.1 registry[2262]: time="2024-06-24T07:51:49.232772614Z" level=info msg="authorized request" go.version=go1.22.3 http.request.host="core:8080" http.request.id=20ceb6d1-c9fb-4c96-bd3b-ddb061f62363 http.request.method=HEAD http.request.remoteaddr=172.28.0.11 http.request.uri="/v2/vergoh/vnstat/manifests/sha256:801dd0ecec9db208f5c84e051dd5dc970080961cb0854c1e9786aa8040a209b1" http.request.useragent=harbor-registry-client vars.name="vergoh/vnstat" vars.reference="sha256:801dd0ecec9db208f5c84e051dd5dc970080961cb0854c1e9786aa8040a209b1" 
Jun 24 07:51:51 172.28.0.1 registry[2262]: 172.28.0.8 - - [24/Jun/2024:07:51:49 +0000] "HEAD /v2/vergoh/vnstat/manifests/sha256:801dd0ecec9db208f5c84e051dd5dc970080961cb0854c1e9786aa8040a209b1 HTTP/1.1" 200 3161 "" "harbor-registry-client"
Jun 24 07:51:51 172.28.0.1 registry[2262]: time="2024-06-24T07:51:51.032964897Z" level=info msg="response completed" go.version=go1.22.3 http.request.host="core:8080" http.request.id=20ceb6d1-c9fb-4c96-bd3b-ddb061f62363 http.request.method=HEAD http.request.remoteaddr=172.28.0.11 http.request.uri="/v2/vergoh/vnstat/manifests/sha256:801dd0ecec9db208f5c84e051dd5dc970080961cb0854c1e9786aa8040a209b1" http.request.useragent=harbor-registry-client http.response.contenttype="application/vnd.oci.image.index.v1+json" http.response.duration=1.806858788s http.response.status=200 http.response.written=3161 
Jun 24 07:51:51 172.28.0.1 registry[2262]: time="2024-06-24T07:51:51.479668802Z" level=info msg="authorized request" go.version=go1.22.3 http.request.host="core:8080" http.request.id=27d3cbb1-3f7f-452a-9313-d5c4e8e6c6c1 http.request.method=HEAD http.request.remoteaddr=172.28.0.11 http.request.uri="/v2/vergoh/vnstat/manifests/sha256:b8ed1b84376584c54d6db232d9d3ac65873c2af117f4eb9b75c2990bd02fc2b2" http.request.useragent=harbor-registry-client vars.name="vergoh/vnstat" vars.reference="sha256:b8ed1b84376584c54d6db232d9d3ac65873c2af117f4eb9b75c2990bd02fc2b2" 
Jun 24 07:51:52 172.28.0.1 registry[2262]: time="2024-06-24T07:51:52.958540222Z" level=info msg="response completed" go.version=go1.22.3 http.request.host="core:8080" http.request.id=27d3cbb1-3f7f-452a-9313-d5c4e8e6c6c1 http.request.method=HEAD http.request.remoteaddr=172.28.0.11 http.request.uri="/v2/vergoh/vnstat/manifests/sha256:b8ed1b84376584c54d6db232d9d3ac65873c2af117f4eb9b75c2990bd02fc2b2" http.request.useragent=harbor-registry-client http.response.contenttype="application/vnd.docker.distribution.manifest.list.v2+json" http.response.duration=1.485548263s http.response.status=200 http.response.written=1415 
Jun 24 07:51:52 172.28.0.1 registry[2262]: 172.28.0.8 - - [24/Jun/2024:07:51:51 +0000] "HEAD /v2/vergoh/vnstat/manifests/sha256:b8ed1b84376584c54d6db232d9d3ac65873c2af117f4eb9b75c2990bd02fc2b2 HTTP/1.1" 200 1415 "" "harbor-registry-client"
Jun 24 07:51:53 172.28.0.1 registry[2262]: time="2024-06-24T07:51:53.35607824Z" level=info msg="authorized request" go.version=go1.22.3 http.request.host="core:8080" http.request.id=66726525-1c6f-462a-ba6d-641087515470 http.request.method=HEAD http.request.remoteaddr=172.28.0.11 http.request.uri="/v2/vergoh/vnstat/manifests/sha256:3ab8b0d77fb181df5cd1ab4a274b97f0e436b1d0e5b7731c897e8d23bc04ad15" http.request.useragent=harbor-registry-client vars.name="vergoh/vnstat" vars.reference="sha256:3ab8b0d77fb181df5cd1ab4a274b97f0e436b1d0e5b7731c897e8d23bc04ad15" 
Jun 24 07:51:57 172.28.0.1 registry[2262]: time="2024-06-24T07:51:57.218205423Z" level=info msg="response completed" go.version=go1.22.3 http.request.host="core:8080" http.request.id=66726525-1c6f-462a-ba6d-641087515470 http.request.method=HEAD http.request.remoteaddr=172.28.0.11 http.request.uri="/v2/vergoh/vnstat/manifests/sha256:3ab8b0d77fb181df5cd1ab4a274b97f0e436b1d0e5b7731c897e8d23bc04ad15" http.request.useragent=harbor-registry-client http.response.contenttype="application/vnd.docker.distribution.manifest.list.v2+json" http.response.duration=3.871801441s http.response.status=200 http.response.written=1415 
Jun 24 07:51:57 172.28.0.1 registry[2262]: 172.28.0.8 - - [24/Jun/2024:07:51:53 +0000] "HEAD /v2/vergoh/vnstat/manifests/sha256:3ab8b0d77fb181df5cd1ab4a274b97f0e436b1d0e5b7731c897e8d23bc04ad15 HTTP/1.1" 200 1415 "" "harbor-registry-client"
Jun 24 07:51:57 172.28.0.1 registry[2262]: 172.28.0.8 - - [24/Jun/2024:07:51:57 +0000] "GET / HTTP/1.1" 200 0 "" "Go-http-client/1.1"
Jun 24 07:51:57 172.28.0.1 registry[2262]: time="2024-06-24T07:51:57.608987118Z" level=info msg="authorized request" go.version=go1.22.3 http.request.host="core:8080" http.request.id=bb5f3f57-f3ed-4de3-9d03-8d79aa0ed747 http.request.method=HEAD http.request.remoteaddr=172.28.0.11 http.request.uri="/v2/vergoh/vnstat/manifests/sha256:79e0e777365d6af717503e09bd909c626d7a379c6d6b2215b283135e2ccddf17" http.request.useragent=harbor-registry-client vars.name="vergoh/vnstat" vars.reference="sha256:79e0e777365d6af717503e09bd909c626d7a379c6d6b2215b283135e2ccddf17" 
Jun 24 07:51:59 172.28.0.1 registry[2262]: 172.28.0.8 - - [24/Jun/2024:07:51:57 +0000] "HEAD /v2/vergoh/vnstat/manifests/sha256:79e0e777365d6af717503e09bd909c626d7a379c6d6b2215b283135e2ccddf17 HTTP/1.1" 200 1415 "" "harbor-registry-client"
Jun 24 07:51:59 172.28.0.1 registry[2262]: time="2024-06-24T07:51:59.087330041Z" level=info msg="response completed" go.version=go1.22.3 http.request.host="core:8080" http.request.id=bb5f3f57-f3ed-4de3-9d03-8d79aa0ed747 http.request.method=HEAD http.request.remoteaddr=172.28.0.11 http.request.uri="/v2/vergoh/vnstat/manifests/sha256:79e0e777365d6af717503e09bd909c626d7a379c6d6b2215b283135e2ccddf17" http.request.useragent=harbor-registry-client http.response.contenttype="application/vnd.docker.distribution.manifest.list.v2+json" http.response.duration=1.484883502s http.response.status=200 http.response.written=1415 
Jun 24 07:52:07 172.28.0.1 registry[2262]: 172.28.0.8 - - [24/Jun/2024:07:52:07 +0000] "GET / HTTP/1.1" 200 0 "" "Go-http-client/1.1"
Jun 24 07:52:08 172.28.0.1 registry[2262]: 127.0.0.1 - - [24/Jun/2024:07:52:08 +0000] "GET / HTTP/1.1" 200 0 "" "curl/8.1.2"
Jun 24 07:52:17 172.28.0.1 registry[2262]: 172.28.0.8 - - [24/Jun/2024:07:52:17 +0000] "GET / HTTP/1.1" 200 0 "" "Go-http-client/1.1"
Jun 24 07:52:27 172.28.0.1 registry[2262]: 172.28.0.8 - - [24/Jun/2024:07:52:27 +0000] "GET / HTTP/1.1" 200 0 "" "Go-http-client/1.1"
Jun 24 07:52:37 172.28.0.1 registry[2262]: 172.28.0.8 - - [24/Jun/2024:07:52:37 +0000] "GET / HTTP/1.1" 200 0 "" "Go-http-client/1.1"
Jun 24 07:52:38 172.28.0.1 registry[2262]: 127.0.0.1 - - [24/Jun/2024:07:52:38 +0000] "GET / HTTP/1.1" 200 0 "" "curl/8.1.2"
Jun 24 07:52:47 172.28.0.1 registry[2262]: 172.28.0.8 - - [24/Jun/2024:07:52:47 +0000] "GET / HTTP/1.1" 200 0 "" "Go-http-client/1.1"

worker.log

2024-06-24T07:51:36Z [INFO] [/controller/replication/transfer/image/transfer.go:139]: client for source registry [type: docker-hub, URL: https://hub.docker.com, insecure: false] created
2024-06-24T07:51:36Z [INFO] [/controller/replication/transfer/image/transfer.go:149]: client for destination registry [type: harbor, URL: http://core:8080, insecure: true] created
2024-06-24T07:51:36Z [INFO] [/controller/replication/transfer/image/transfer.go:182]: copying vergoh/vnstat:[latest,2.12,dev,2.11,2.10,2.9,2.8,2.7](source registry) to vergoh/vnstat:[latest,2.12,dev,2.11,2.10,2.9,2.8,2.7](destination registry)...
2024-06-24T07:51:36Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying vergoh/vnstat:latest(source registry) to vergoh/vnstat:latest(destination registry)...
2024-06-24T07:51:36Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact vergoh/vnstat:latest ...
2024-06-24T07:51:40Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact vergoh/vnstat:latest pulled
2024-06-24T07:51:44Z [INFO] [/controller/replication/transfer/image/transfer.go:226]: the artifact vergoh/vnstat:latest already exists on the destination registry, skip
2024-06-24T07:51:44Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying vergoh/vnstat:2.12(source registry) to vergoh/vnstat:2.12(destination registry)...
2024-06-24T07:51:44Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact vergoh/vnstat:2.12 ...
2024-06-24T07:51:45Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact vergoh/vnstat:2.12 pulled
2024-06-24T07:51:45Z [INFO] [/controller/replication/transfer/image/transfer.go:226]: the artifact vergoh/vnstat:2.12 already exists on the destination registry, skip
2024-06-24T07:51:45Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying vergoh/vnstat:dev(source registry) to vergoh/vnstat:dev(destination registry)...
2024-06-24T07:51:45Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact vergoh/vnstat:dev ...
2024-06-24T07:51:45Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact vergoh/vnstat:dev pulled
2024-06-24T07:51:46Z [INFO] [/controller/replication/transfer/image/transfer.go:226]: the artifact vergoh/vnstat:dev already exists on the destination registry, skip
2024-06-24T07:51:46Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying vergoh/vnstat:2.11(source registry) to vergoh/vnstat:2.11(destination registry)...
2024-06-24T07:51:46Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact vergoh/vnstat:2.11 ...
2024-06-24T07:51:47Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact vergoh/vnstat:2.11 pulled
2024-06-24T07:51:48Z [INFO] [/controller/replication/transfer/image/transfer.go:226]: the artifact vergoh/vnstat:2.11 already exists on the destination registry, skip
2024-06-24T07:51:48Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying vergoh/vnstat:2.10(source registry) to vergoh/vnstat:2.10(destination registry)...
2024-06-24T07:51:48Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact vergoh/vnstat:2.10 ...
2024-06-24T07:51:49Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact vergoh/vnstat:2.10 pulled
2024-06-24T07:51:51Z [INFO] [/controller/replication/transfer/image/transfer.go:226]: the artifact vergoh/vnstat:2.10 already exists on the destination registry, skip
2024-06-24T07:51:51Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying vergoh/vnstat:2.9(source registry) to vergoh/vnstat:2.9(destination registry)...
2024-06-24T07:51:51Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact vergoh/vnstat:2.9 ...
2024-06-24T07:51:51Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact vergoh/vnstat:2.9 pulled
2024-06-24T07:51:52Z [INFO] [/controller/replication/transfer/image/transfer.go:226]: the artifact vergoh/vnstat:2.9 already exists on the destination registry, skip
2024-06-24T07:51:52Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying vergoh/vnstat:2.8(source registry) to vergoh/vnstat:2.8(destination registry)...
2024-06-24T07:51:52Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact vergoh/vnstat:2.8 ...
2024-06-24T07:51:53Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact vergoh/vnstat:2.8 pulled
2024-06-24T07:51:57Z [INFO] [/controller/replication/transfer/image/transfer.go:226]: the artifact vergoh/vnstat:2.8 already exists on the destination registry, skip
2024-06-24T07:51:57Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying vergoh/vnstat:2.7(source registry) to vergoh/vnstat:2.7(destination registry)...
2024-06-24T07:51:57Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact vergoh/vnstat:2.7 ...
2024-06-24T07:51:57Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact vergoh/vnstat:2.7 pulled
2024-06-24T07:51:59Z [INFO] [/controller/replication/transfer/image/transfer.go:226]: the artifact vergoh/vnstat:2.7 already exists on the destination registry, skip
2024-06-24T07:51:59Z [INFO] [/controller/replication/transfer/image/transfer.go:204]: copy vergoh/vnstat:[latest,2.12,dev,2.11,2.10,2.9,2.8,2.7](source registry) to vergoh/vnstat:[latest,2.12,dev,2.11,2.10,2.9,2.8,2.7](destination registry) completed
MinerYang commented 5 months ago

How many tags approximately in your repository when hits the 429 rate limit ?

BrettDean commented 5 months ago

About 398 tags will consume 200 quotas, I suppose. Because this image has 335 tags, using up 168 quotas:

proxy.log

https://raw.githubusercontent.com/BrettDean/temp/main/20240624/proxy.log

registry.log

https://raw.githubusercontent.com/BrettDean/temp/main/20240624/registry.log

worker.log

https://raw.githubusercontent.com/BrettDean/temp/main/20240624/worker.log

quota_remaining.log https://raw.githubusercontent.com/BrettDean/temp/main/20240624/quota_remaining.log

stonezdj commented 5 months ago

The GET API for tags/list consumes quota, the max page_size is 100 in dockerhub api /v2/<repository_name>/tags/list, there is no room to reduce the quota usage.

BrettDean commented 5 months ago

Thank you for your reply.

I believe get the tags list does not require quota consumption, as shown in the script below. We can even get the tag list without logging into Docker Hub:

#!/bin/bash

userRepo=$1
tag_count=0

function list_tags() {
    local page_num=1
    local page_size=100

    while true; do
        local tags_json=$(curl -s "https://registry.hub.docker.com/v2/repositories/${userRepo}/tags/?page=${page_num}&page_size=${page_size}")

        if [[ $(echo "${tags_json}" | jq -r '.results | length') -eq 0 ]]; then
            break
        fi

        local tags=$(echo "${tags_json}" | jq -r '.results[].name')
        local tags_count=$(echo "${tags_json}" | jq -r '.count')
        tag_count=${tags_count}

        echo "${tags}"

        ((page_num++))
    done
}

list_tags "${userRepo}"
echo "There're ${tag_count} tags in ${userRepo}"

run:

bash printTags.sh jc21/nginx-proxy-manager

It'll return the list of tags:

https://raw.githubusercontent.com/BrettDean/temp/main/20240625/printTags.log

all_data:

https://raw.githubusercontent.com/BrettDean/temp/main/20240625/all_data_it_returns.json

It won’t count towards the quota of the current IP anonymous user. After running the script to get the tag list, the quota still remains 100.

ratelimit-remaining: 100;w=21600

shishir-11 commented 5 months ago

Hi, I'm hoping to contribute to harbor and am a bit new to the field and had some question, after setting up my development environment how do i run and test the functionality of harbor and changes I make ?

github-actions[bot] commented 3 months ago

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

adrianmoisey commented 3 months ago

Bumping this, it would be nice to have

github-actions[bot] commented 2 weeks ago

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.