Garbage collection running wild

danielbecker commented 3 years ago

Description

Since old tags for our images were piling up, I enabled garbage collection to run with keep_latest: 10 and older_than: 90. From the documentation, I assume this means remove all tags but the last 10 and don't consider any tag younger than 90 days.

However, what happened was a ton of old, admin and template images being removed entirely. The images weren't updated recently, but frequently pulled, as we need them in our daily deployment process.

Steps to reproduce

Enable garbage collection:
- PORTUS_DELETE_GARBAGE_COLLECTOR_ENABLED=true
- PORTUS_DELETE_GARBAGE_COLLECTOR_OLDER_THAN=90
- PORTUS_DELETE_GARBAGE_COLLECTOR_KEEP_LATEST=10
Restart portus docker-compose up -d
Watch your images diappear

Expected behavior: Remove tags older than 90 days and keep at least 10 around
Actual behavior: Entire images were removed

Providing logs of the moment when the issue has happened would also be useful. If you are in production, you might want to set the PORTUS_LOG_LEVEL to debug to get a more verbose log.

Deployment information

Deployment method: docker-compose.yml

version: "3.7"

services: portus: image: opensuse/portus:2.4 restart: unless-stopped environment:

PORTUS_MACHINE_FQDN_VALUE=xxx

DB. The password for the database should definitely not be here. You are

probably better off with Docker Swarm secrets.
PORTUS_DB_HOST=db
PORTUS_DB_DATABASE=xxx
PORTUS_DB_PASSWORD=xxx
PORTUS_DB_POOL=5

Secrets. It can possibly be handled better with Swarm's secrets.
PORTUS_SECRET_KEY_BASE=xxx
PORTUS_KEY_PATH=/certificates/2019_key.pem
PORTUS_PASSWORD=xxx

SSL
PORTUS_PUMA_TLS_KEY=/certificates/2019_key.pem
PORTUS_PUMA_TLS_CERT=/certificates/2019_cert.pem
RAILS_SERVE_STATIC_FILES=false ports:
127.0.0.1:8063:3000 links:
db volumes:
/srv/xxx/portus/certs:/certificates:ro
/srv/xxx/portus/assets:/srv/Portus/public extra_hosts:
"xxx:192.168.252.113" logging: driver: "gelf" options: gelf-address: "tcp://xxx:5555" tag: "portus-frontend" mode: non-blocking max-buffer-size: 1g

background: image: opensuse/portus:2.4 restart: unless-stopped environment:
CCONFIG_PREFIX=PORTUS
PORTUS_MACHINE_FQDN_VALUE=xxx

DB. The password for the database should definitely not be here. You are

probably better off with Docker Swarm secrets.
PORTUS_DB_HOST=db
PORTUS_DB_DATABASE=xxx
PORTUS_DB_PASSWORD=xxx
PORTUS_DB_POOL=5

Secrets. It can possibly be handled better with Swarm's secrets.
PORTUS_SECRET_KEY_BASE=xxx
PORTUS_KEY_PATH=/certificates/2019_key.pem
PORTUS_PASSWORD=xxx

Enable garbage collection
PORTUS_DELETE_GARBAGE_COLLECTOR_ENABLED=true
PORTUS_DELETE_GARBAGE_COLLECTOR_OLDER_THAN=90
PORTUS_DELETE_GARBAGE_COLLECTOR_KEEP_LATEST=10
PORTUS_BACKGROUND=true links:
db volumes:
/srv/xxx/portus/certs:/certificates:ro extra_hosts:
"xxx:192.168.252.113" logging: driver: "gelf" options: gelf-address: "tcp://xxx:5555" tag: "portus-backend" mode: non-blocking max-buffer-size: 1g

db: image: library/mariadb:10.0.23 restart: unless-stopped command: mysqld --character-set-server=utf8 --collation-server=utf8_unicode_ci --init-connect='SET NAMES UTF8;' --innodb-flush-log-at-trx-commit=0 environment:
MYSQL_DATABASE=xxx
MYSQL_ROOT_PASSWORD=xxx volumes:
/srv/xxx/portus/database:/var/lib/mysql

registry: image: library/registry:2.6 restart: unless-stopped command: ["/bin/sh", "/etc/docker/registry/init"] environment:

Authentication

REGISTRY_AUTH_TOKEN_REALM: https://xxx/v2/token REGISTRY_AUTH_TOKEN_SERVICE: xxx REGISTRY_AUTH_TOKEN_ISSUER: xxx REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE: /secrets/2019_bundle.pem

SSL

REGISTRY_HTTP_TLS_CERTIFICATE: /secrets/2019_cert.pem REGISTRY_HTTP_TLS_KEY: /secrets/2019_key.pem

Portus endpoint

REGISTRY_NOTIFICATIONS_ENDPOINTS: >
- name: portus url: https://xxx/v2/webhooks/events timeout: 2000ms threshold: 5 backoff: 1s volumes:
/srv/xxx/portus/registry/storage:/var/lib/registry
/srv/xxx/portus/certs:/secrets:ro
/srv/xxx/portus/registry/config/config.yml:/etc/docker/registry/config.yml:ro
/srv/xxx/portus/registry/init/init:/etc/docker/registry/init:ro ports:
127.0.0.1:xxx:5000
127.0.0.1:xxx:5001 # required to access debug service links:
portus:portus logging: driver: "gelf" options: gelf-address: "tcp://xxx:5555" tag: "portus-registry" mode: non-blocking max-buffer-size: 1g

Configuration:

Please note that I had to turn off garbage collection for now to stop portus from causing more damage.

[Mailer config] Host: portus.test.lan [Mailer config] Protocol: https:// Evaluated configuration:

email: from: portus@example.com name: Portus reply_to: '' smtp: enabled: false address: smtp.example.com port: 587 domain: example.com ssl_tls: '' enable_starttls_auto: false openssl_verify_mode: none ca_path: '' ca_file: '' user_name: '' password: "" authentication: login gravatar: enabled: true delete: enabled: true contributors: false garbage_collector: enabled: false older_than: 30 tag: '' ldap: enabled: false hostname: ldap_hostname port: 389 timeout: 5 encryption: method: '' options: ca_file: '' ssl_version: TLSv1_2 base: '' filter: '' uid: uid authentication: enabled: false bind_dn: '' password: "" guess_email: enabled: false attr: '' oauth: local_login: enabled: true google_oauth2: enabled: false id: '' secret: '' domain: '' options: hd: '' open_id: enabled: false identifier: '' domain: '' github: enabled: false client_id: '' client_secret: '' organization: '' team: '' domain: '' gitlab: enabled: false application_id: '' secret: '' group: '' domain: '' server: '' bitbucket: enabled: false key: '' secret: '' domain: '' options: team: '' first_user_admin: enabled: true signup: enabled: true check_ssl_usage: enabled: true registry: jwt_expiration_time: value: 15 catalog_page: value: 100 timeout: value: 2 read_timeout: value: 120 machine_fqdn: value: portus.easybell.de display_name: enabled: false user_permission: change_visibility: enabled: true create_team: enabled: true manage_team: enabled: true create_namespace: enabled: true manage_namespace: enabled: true create_webhook: enabled: true manage_webhook: enabled: true push_images: policy: allow-teams security: clair: server: '' health_port: 6061 timeout: 900 zypper: server: '' dummy: server: '' anonymous_browsing: enabled: true background: registry: enabled: true sync: enabled: true strategy: initial pagination: limit: 10 per_page: 10 before_after: 2

Portus version: 2.4.3@5a616c0ef860567df5700708256f42505cdb9952

version: 0.1
storage:
  filesystem:
    rootdirectory: /var/lib/registry
  delete:
    enabled: true
http:
  addr: 0.0.0.0:5000
  debug:
    addr: 0.0.0.0:5001

Portus version: 2.4.3@5a616c0ef860567df5700708256f42505cdb9952

Bonko commented 3 years ago

As far as i know this feature was planned for Portus 2.5: https://github.com/SUSE/Portus/issues/2051

Sadly the documentation refers to master instead of current stable

stale[bot] commented 2 years ago

Thanks for all your contributions! This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

SUSE / Portus