Garbage collector not clean redis keys

[nevlkv]

Expected behavior and actual behavior: Garbage collector not cleanup redis keys, and failed with message

2023-09-06T02:37:23Z [INFO] [/jobservice/job/impl/gc/garbage_collection.go:414]: 2937 blobs and 812 manifests are actually deleted 2023-09-06T02:37:23Z [INFO] [/jobservice/job/impl/gc/garbage_collection.go:415]: The GC job actual frees up 54343 MB space. 2023-09-06T02:37:23Z [ERROR] [/jobservice/job/impl/gc/garbage_collection.go:445]: failed to clean registry cache error retrieving 'blobs::*' keys: WRONGPASS invalid username-password pair or user is disabled., pattern blobs::*

Steps to reproduce the problem:

Starts registry cleanup

Versions:

• harbor version: 2.8.4

Additional context:

external redis with default user

_REDIS_URL_REG: redis://:password@harbor-redis-master:6379/2?idle_timeout_seconds=30

[wy65701436]

the error points the auth info is incorrect, can you check that at your end?

[nevlkv]

password correct

redis-cli -n 2 -a <password>
127.0.0.1:6379[2]> info keyspace
# Keyspace
db0:keys=133200,expires=133085,avg_ttl=2212096
db1:keys=2036,expires=1748,avg_ttl=83530815
db2:keys=5784,expires=156,avg_ttl=85886397
db5:keys=63,expires=54,avg_ttl=2360250

[nevlkv]

garbage service used _REDIS_URL_REG: ?

[MinerYang]

Hi @nevlkv

• Is this a fresh install harbor? Have you changed the redis password after installation?
• Could you also check/provide other redis url, for example _REDIS_URL_CORE?

[nevlkv]

1. current instance was upgraded from 2.7.1, redis is external and not changed
2. we use default redis account and password always identical

   
   kubectl get cm harbor-core -n=io  -o yaml
   apiVersion: v1
   data:
   _REDIS_URL_CORE: redis://:<password>@harbor-redis-master:6379/0?idle_timeout_seconds=30
   _REDIS_URL_REG: redis://:<password>@harbor-redis-master:6379/2?idle_timeout_seconds=30

i can auth with <password> via redis-cli and flushdb

GC [use same connection string](https://github.com/goharbor/harbor/blob/v2.8.4/src/jobservice/job/impl/gc/garbage_collection.go#L114)

[viceice]

Seeing same error:

023-09-17T00:02:59Z [ERROR] [/jobservice/job/impl/gc/garbage_collection.go:515]: failed to clean registry cache failed to scan keys: WRONGPASS invalid username-password pair or user is disabled., pattern blobs::*

I've no password for my external redis, i'm using network policies to limit redis access to harbor.

_REDIS_URL_CORE: redis://keydb:6379/0?idle_timeout_seconds=30
_REDIS_URL_REG: redis://keydb:6379/2?idle_timeout_seconds=30

Harbor version: v2.9.0

[viceice]

It seems the redis url is not passed to the job service via env.

I only see this inside config.yml if job service:

worker_pool:
  workers: 10
  backend: "redis"
  redis_pool:
    redis_url: "redis://keydb:6379/1"
    namespace: "harbor_job_service_namespace"
    idle_timeout_second: 3600

[viceice]

We probably need to set ?

_REDIS_URL_REG: "{{ template "harbor.redis.urlForRegistry" . }}"

https://github.com/goharbor/harbor-helm/blob/86dd759ac03e9c781c25a6c0d71ec8d95368fb32/templates/jobservice/jobservice-cm-env.yaml#L15

[nevlkv]

error stable

2023-09-21T07:55:57Z [ERROR] [/jobservice/runner/redis.go:123]: Job 'GARBAGE_COLLECTION:c09d285cf0794f361744acab' exit with error: run error: error retrieving 'blobs::*' keys: WRONGPASS invalid username-password pair or user is disabled.

[nevlkv]

Current state

2023-09-24T02:31:20Z [INFO] [/jobservice/job/impl/gc/garbage_collection.go:414]: 3177 blobs and 562 manifests are actually deleted
2023-09-24T02:31:20Z [INFO] [/jobservice/job/impl/gc/garbage_collection.go:415]: The GC job actual frees up 37506 MB space.
2023-09-24T02:31:20Z [ERROR] [/jobservice/job/impl/gc/garbage_collection.go:445]: failed to clean registry cache error retrieving 'blobs::*' keys: WRONGPASS invalid username-password pair or user is disabled., pattern blobs::*

[MinerYang]

Hi @nevlkv , Could you try to monitor the registry log when pushing images, if there's any error msg?

[nevlkv]

Sometimes on pull 404 with "blob unknown"

time="2023-09-26T11:19:16.21575386Z" level=error msg="response completed with error" auth.user.name=harbor err.code="blob unknown" err.detail=sha256:7dbc1adf280e1aa588c033eaa746aa6db327ee16be705740f81741f5e6945c86 err.message="blob unknown to registry" go.version=go1.20.7 http.request.host=REGISTRY http.request.id=cdc0816e-2548-40d7-b33c-3f2f9a26837b http.request.method=HEAD http.request.remoteaddr=*.*.*.* http.request.uri="/v2/REPO/blobs/sha256:7dbc1adf280e1aa588c033eaa746aa6db327ee16be705740f81741f5e6945c86" http.request.useragent="docker/20.10.22 go/go1.18.9 git-commit/42c8b31 kernel/5.15.0-83-generic os/linux arch/amd64 UpstreamClient(docker-compose/1.29.2 docker-py/6.1.3 Linux/5.15.0-83-generic)" http.response.contenttype="application/json; charset=utf-8" http.response.duration=104.591881ms http.response.status=404 http.response.written=157 vars.digest="sha256:7dbc1adf280e1aa588c033eaa746aa6db327ee16be705740f81741f5e6945c86" vars.name="REPO"

[nevlkv]

If default user in URI undefined rise error

$ redis-cli -u redis://:<password>@harbor-redis-master:6379/2?idle_timeout_seconds=30 info keyspace
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
AUTH failed: WRONGPASS invalid username-password pair or user is disabled.
$ redis-cli -u redis://default:<password>@harbor-redis-master:6379/2?idle_timeout_seconds=30 info keyspace 
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
# Keyspace
db0:keys=1732,expires=1732,avg_ttl=1826800
db1:keys=116,expires=65,avg_ttl=44559991
db2:keys=1,expires=0,avg_ttl=0
db5:keys=4,expires=0,avg_ttl=0

define user in values

   type: external
   external:
     addr: "harbor-redis-master:6379"
+    username: "default"

and

kubectl get cm harbor-core -n=io  -o yaml
apiVersion: v1
data:
  _REDIS_URL_CORE: redis://:<password>@harbor-redis-master:6379/0?idle_timeout_seconds=30
  _REDIS_URL_REG: redis://:<password>@harbor-redis-master:6379/2?idle_timeout_seconds=30

became

kubectl get cm harbor-core -n=io  -o yaml
apiVersion: v1
data:
  _REDIS_URL_CORE: redis://default:<password>@harbor-redis-master:6379/0?idle_timeout_seconds=30
  _REDIS_URL_REG: redis://default:<password>@harbor-redis-master:6379/2?idle_timeout_seconds=30

and GC success !!!

[viceice]

Interestingly a manual run works, but the sheduled runs are always failing. 😕

[nevlkv]

same error after upgrade to 2.10

scheduled fail, manual success

upgraded 08/02/2024

[github-actions[bot]]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[viceice]

not stale 😞

[nevlkv]

successful only manual job

[nevlkv]

execute success only manual

[nevlkv]

If GC scheduled and settings of harbor changed after, job became failed:

registry=# select * from schedule where callback_func_name='GARBAGE_COLLECTION'
registry-# ;
  id  |       creation_time        |        update_time         |    vendor_type     | vendor_id |    cron     | callback_func_name |                                                                                                         callback_func_param                                                                                                          | cron_type |       extra_attrs        |  revision  
------+----------------------------+----------------------------+--------------------+-----------+-------------+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+--------------------------+------------
 1009 | 2022-07-21 10:09:37.790778 | 2022-07-21 10:09:37.790778 | GARBAGE_COLLECTION |        -1 | 0 0 0 * * * | GARBAGE_COLLECTION | {"trigger":null,"deleteuntagged":true,"dryrun":false,"extra_attrs":{"delete_untagged":true,"dry_run":false,"redis_url_reg":"redis://redis:<any_password>@harbor-redis-master:6379/2?idle_timeout_seconds=30","time_window":2}} | Daily     | {"delete_untagged":true} | 1717372800
(1 row)

redis_url_reg static and not changes!

[github-actions[bot]]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[wy65701436]

@nevlkv can you share your settings when installing your harbor? You are seeing the error only happen at the scheduled GC, right?