search

goharbor / harbor

An open source trusted cloud native registry project that stores, signs, and scans content.
https://goharbor.io
Apache License 2.0
24.09k stars 4.76k forks source link

v1.10.0 Garbage Collection is not working in my environment #10503

Closed joeyjy closed 4 years ago

joeyjy commented 4 years ago

Hi Harbor Team,

Garbage Collection is not working in my fresh installed environment.

Expected behavior and actual behavior: Click GC NOW button in Garbage Collection page, images' data in the storage backend(Ceph) will be removed

Steps to reproduce the problem:

  1. Delete repository in repositories page.
  2. Click GC NOW button in Garbage Collection page, says Garbage Collection Successful.
  3. List data in storage backend(Ceph) use swift command and find nothing changed(total size, object number...)
  4. See the Garbage Collection History(http://harbor.host/api/system/gc/1/log), return 404.

Versions:

Additional context:

# docker-compose ps
      Name                     Command                  State                 Ports
---------------------------------------------------------------------------------------------
harbor-core         /harbor/harbor_core              Up (healthy)
harbor-jobservice   /harbor/harbor_jobservice  ...   Up (healthy)
harbor-log          /bin/sh -c /usr/local/bin/ ...   Up (healthy)   127.0.0.1:1514->10514/tcp
harbor-portal       nginx -g daemon off;             Up (healthy)   8080/tcp
nginx               nginx -g daemon off;             Up (healthy)   0.0.0.0:80->8080/tcp
registry            /home/harbor/entrypoint.sh       Up (healthy)   5000/tcp
registryctl         /home/harbor/start.sh            Up (healthy)

The IP address or hostname to access admin UI and registry service.

DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.

hostname: harbor.cloud.mycorp.com

http related config

http:

port for http, default is 80. If https enabled, this port will redirect to https port

port: 80

https related config

https:

https port for harbor, default is 443

port: 443

The path of cert and key files for nginx

certificate: /your/certificate/path

private_key: /your/private/key/path

Uncomment external_url if you want to enable external proxy

And when it enabled the hostname will no longer used

external_url: https://reg.mydomain.com:8433

The initial password of Harbor admin

It only works in first time to install harbor

Remember Change the admin password from UI after launching Harbor.

harbor_admin_password: BS5p23Xk7CubcjjuTkgJ

Harbor DB configuration

database:

The password for the root user of Harbor DB. Change this before any production use.

password: root123

The maximum number of connections in the idle connection pool. If it <=0, no idle connections are retained.

max_idle_conns: 50

The maximum number of open connections to the database. If it <= 0, then there is no limit on the number of open connections.

Note: the default number of connections is 100 for postgres.

max_open_conns: 100

The default data volume

data_volume: /data

Harbor Storage settings by default is using /data dir on local filesystem

Uncomment storage_service setting If you want to using external storage

storage_service: swift: username: xxx password: xxx authurl: http://keystone.cloud.mycorp.com:5000/v2.0 tenant: service domain: harbor.cloud.mycorp.com region: Regionxxx container: xxxxxx cache: layerinfo: inmemory maintenance: uploadpurging: enabled: false delete: enabled: true

ca_bundle is the path to the custom root ca certificate, which will be injected into the truststore

of registry's and chart repository's containers. This is usually needed when the user hosts a internal storage with self signed certificate.

ca_bundle:

storage backend, default is filesystem, options include filesystem, azure, gcs, s3, swift and oss

for more info about this configuration please refer https://docs.docker.com/registry/configuration/

filesystem:

maxthreads: 100

set disable to true when you want to disable registry redirect

redirect:

disabled: false

Clair configuration

clair:

The interval of clair updaters, the unit is hour, set to 0 to disable the updaters.

updaters_interval: 12

jobservice:

Maximum number of job workers in job service

max_job_workers: 10

notification:

Maximum retry count for webhook job

webhook_job_max_retry: 10

chart:

Change the value of absolute_url to enabled can enable absolute url in chart

absolute_url: disabled

Log configurations

log:

options are debug, info, warning, error, fatal

level: info

configs for logs in local storage

local:

Log files are rotated log_rotate_count times before being removed. If count is 0, old versions are removed rather than rotated.

rotate_count: 50
# Log files are rotated only if they grow bigger than log_rotate_size bytes. If size is followed by k, the size is assumed to be in kilobytes.
# If the M is used, the size is in megabytes, and if G is used, the size is in gigabytes. So size 100, size 100k, size 100M and size 100G
# are all valid.
rotate_size: 200M
# The directory on your host that store log
location: /var/log/harbor

Uncomment following lines to enable external syslog endpoint.

external_endpoint:

protocol used to transmit log to external endpoint, options is tcp or udp

protocol: tcp

The host of external endpoint

host: localhost

Port of external endpoint

port: 5140

This attribute is for migrator to detect the version of the .cfg file, DO NOT MODIFY!

_version: 1.10.0

Uncomment external_database if using external database.

external_database: harbor: host: 10.61.xx.xx port: 5432 db_name: harbor_db username: xxxxx password: xxxxx ssl_mode: disable max_idle_conns: 50 max_open_conns: 0 clair: host: 10.61.xx.xx port: 5432 db_name: clair_db username: xxxxx password: xxxxx ssl_mode: disable notary_signer: host: 10.61.xx.xx port: 5432 db_name: notary_signer_db username: xxxxx password: xxxxx ssl_mode: disable notary_server: host: 10.61.xx.xx port: 5432 db_name: notary_server_db username: xxxxx password: xxxxx ssl_mode: disable

Uncomment external_redis if using external Redis server

external_redis: host: 10.62.xx.xx port: 6379

password:

db_index 0 is for core, it's unchangeable

registry_db_index: 1 jobservice_db_index: 2 chartmuseum_db_index: 3 clair_db_index: 4

Uncomment uaa for trusting the certificate of uaa instance that is hosted via self-signed cert.

uaa:

ca_file: /path/to/ca

Global proxy

Config http proxy for components, e.g. http://my.proxy.com:3128

Components doesn't need to connect to each others via http proxy.

Remove component from components array if want disable proxy

for it. If you want use proxy for replication, MUST enable proxy

for core and jobservice, and set http_proxy and https_proxy.

Add domain to the no_proxy field, when you want disable proxy

for some special registry.

proxy: http_proxy: https_proxy:

no_proxy endpoints will appended to 127.0.0.1,localhost,.local,.internal,log,db,redis,nginx,core,portal,postgresql,jobservice,registry,registryctl,clair,chartmuseum,notary-server

no_proxy: components:

Log files: jobservice log:

[ERROR] [/jobservice/api/handler.go:288]: Serve http request 'GET /api/v1/jobs/220214771e2ab212d4cfdd54/log' error: 404 {"code":10010,"message":"object is not found","details":"220214771e2ab212d4cfdd54"}

core.log:

201 #033[0m|   6.373202ms|   match|#033[46m POST    #033[0m /api/system/gc/schedule   r:/api/system/gc/schedule#033[0m

Only "GET /api/health HTTP/1.1" 200 in registryctl.log

I can find key {harbor_job_service_namespace}:job_stats:220214771e2ab212d4cfdd54 in redis db 2

Need your help

wy65701436 commented 4 years ago

can you refer to https://github.com/goharbor/harbor/issues/7945#issuecomment-498123934 and provide results?

joeyjy commented 4 years ago

@wy65701436 seems to work well:

......
testjunyiyang/harbor-db
testjunyiyang/harbor-db: marking manifest sha256:6dbbd03abbd8d18f2692ba8f89fdeb55cf18d41029aefcce47b522cf3ea967c2
testjunyiyang/harbor-db: marking blob sha256:634404a417cf430406ca5089bf8e30412339b452b03504a467a254b3ef2f68f5
testjunyiyang/harbor-db: marking blob sha256:b950b5dd94ab91474e984c0d6cddfe20dd9437325869b2ce1bf3b48749cdf838
testjunyiyang/harbor-db: marking blob sha256:a66af5ebde941e82219e10dd9aee0fe75a3cf6462df12aa884b79b6dbafccbff
testjunyiyang/harbor-db: marking blob sha256:05611b2cb74d4fab9986e75e63e8707fd2e7e4939a197a873a449cd17b90f030
testjunyiyang/harbor-db: marking blob sha256:c2ec687f77958f30dbfcee7aed2954e866de51a9579007377010d8028bb8471e
testjunyiyang/harbor-db: marking blob sha256:a516bbf5835bb5b96518af0c489bc8c0dbd50c55501324c1133e824ec2bc85c7
testjunyiyang/harbor-db: marking blob sha256:eac82587810bf9008f3efc175fe25cfea080b9a4e18ceaf5019acf57dd0a9e5b
testjunyiyang/harbor-db: marking blob sha256:5cdc9f9fbaab0d0ace47723c2031d8ef37ce96f1e4a9873daf28aa55ac4fea88
testjunyiyang/harbor-db: marking blob sha256:b97542249e3dc3594169709b1d1be3797fe59b535adde6347dee536e728ced5d
testjunyiyang/harbor-db: marking blob sha256:cb1d1ff4115782935286336ca911f825438726fbd4f419ea2711f0947196f494
manifest eligible for deletion: sha256:ac1743a816cee0a83291067e5719393933af369a6d1090bd67945b1cda738ec4
INFO[0001] deleting manifest tag reference: /docker/registry/v2/repositories/testjunyiyang/harbor-db/_manifests/tags/v1.10.0/index/sha256/ac1743a816cee0a83291067e5719393933af369a6d1090bd67945b1cda738ec4  go.version=go1.12.12 instance.id=69a17aef-129b-4f91-a124-032d794cd3d7 service=registry
INFO[0001] deleting manifest: /docker/registry/v2/repositories/testjunyiyang/harbor-db/_manifests/revisions/sha256/ac1743a816cee0a83291067e5719393933af369a6d1090bd67945b1cda738ec4  go.version=go1.12.12 instance.id=69a17aef-129b-4f91-a124-032d794cd3d7 service=registry
24 blobs marked, 54 blobs and 1 manifests eligible for deletion
blob eligible for deletion: sha256:1cb5921864bf77eaeff388e74bfb2d6516a3afc3b98ab47e20da8c21aff719a7
INFO[0007] Deleting blob: /docker/registry/v2/blobs/sha256/1c/1cb5921864bf77eaeff388e74bfb2d6516a3afc3b98ab47e20da8c21aff719a7  go.version=go1.12.12 instance.id=69a17aef-129b-4f91-a124-032d794cd3d7 service=registry
blob eligible for deletion: sha256:2bdc91d6afee063a89c42c542046b4dbf1120ac6633fb6ee4f5ea2d54f9cc351
INFO[0007] Deleting blob: /docker/registry/v2/blobs/sha256/2b/2bdc91d6afee063a89c42c542046b4dbf1120ac6633fb6ee4f5ea2d54f9cc351  go.version=go1.12.12 instance.id=69a17aef-129b-4f91-a124-032d794cd3d7 service=registry
......

any idea?

Thanks!

wy65701436 commented 4 years ago

can you see the log changes in the directory of /data/job_logs? If you trigger a GC job, there should be a new log file generated.

odeng888 commented 4 years ago

I have the same problem!!!

I run command: sudo -E -u #10000 registry garbage-collect --delete-untagged=true /etc/registry/config.yml

get error: failed to garbage collect: failed to mark: s3aws: Path not found: /docker/registry/v2/repositories

Harbor Version: 1.9.0 the storage backend is Ceph.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.