search

SUSE / Portus

Authorization service and frontend for Docker registry (v2)
http://port.us.org/
Apache License 2.0
3k stars 471 forks source link

Image successfully pushed can not be found in the webinterface #2265

Closed danielbecker closed 4 years ago

danielbecker commented 4 years ago

Description

It seems like our portus installation has disconnected from the registry, even though the link icon is still green on the registry page. However, newly pushed repositories and tags are not showing up in the web interface. It is accessible through the registry. So the upload definitely worked.

Steps to reproduce

  1. Push an image to the registry
  2. Go to the webinterface
  3. The image is not showing

In addition, I'm seeing this error message on the registry every second:

time="2019-12-04T16:06:18Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.7.6 http.request.host=portus.xxx http.request.id=4f41dc46-96f6-4f3e-8a19-869b2370674c http.request.method=GET http.request.remoteaddr=10.10.99.4 http.request.uri="/v2/deploy/xxx/manifests/sha256:cdf70ac66018503de92cea923a5b14f38e9f9362c7d6803086f18d055a923191" http.request.useragent=Ruby instance.id=1c81fe94-e4af-4642-a019-a17898ba0145 vars.name="deploy/xxx" vars.reference="sha256:cdf70ac66018503de92cea923a5b14f38e9f9362c7d6803086f18d055a923191" version=v2.6.2-14-ga66a4c3

Deployment information

The deployment is done through docker-compose with an nginx reverse proxy in front of the setup.

Configuration:

You can get this information like this:

version: "3.7"

services:
  portus:
    image: opensuse/portus:2.4
    restart: unless-stopped
    environment:
      - PORTUS_MACHINE_FQDN_VALUE=portus.xxx

      # DB. The password for the database should definitely not be here. You are
      # probably better off with Docker Swarm secrets.
      - PORTUS_DB_HOST=db
      - PORTUS_DB_DATABASE=portus_production
      - PORTUS_DB_PASSWORD=MKFuJvPZvjhp7apE
      - PORTUS_DB_POOL=5

      # Secrets. It can possibly be handled better with Swarm's secrets.
      - PORTUS_SECRET_KEY_BASE=b4a45d52e329d70f5e05893ced9f92575a3e85b74f6a35a8596c0739ff764fda71013ea2fe885f6510d75f80a8b5f8b1f44485f8b062cefe00766e545a60e7fa
      - PORTUS_KEY_PATH=/certificates/2019_key.pem
      - PORTUS_PASSWORD=J2o3ddRVnePRLYxU

      # SSL
      - PORTUS_PUMA_TLS_KEY=/certificates/2019_key.pem
      - PORTUS_PUMA_TLS_CERT=/certificates/2019_cert.pem

      - RAILS_SERVE_STATIC_FILES=false
    ports:
      - 127.0.0.1:8063:3000
    links:
      - db
    volumes:
      - /srv/persistence/portus/certs:/certificates:ro
      - /srv/persistence/portus/assets:/srv/Portus/public
    extra_hosts:
      - "portus.xxx:192.168.252.113"
    deploy:
      placement:
        constraints: [node.hostname == srv-007]
    logging:
      driver: "gelf"
      options:
        gelf-address: "tcp://graylog.xxx:5555"
        tag: "portus-frontend"
        mode: non-blocking
        max-buffer-size: 1g

  background:
    image: opensuse/portus:2.4
    restart: unless-stopped
    environment:
      - CCONFIG_PREFIX=PORTUS
      - PORTUS_MACHINE_FQDN_VALUE=portus.xxx

      # DB. The password for the database should definitely not be here. You are
      # probably better off with Docker Swarm secrets.
      - PORTUS_DB_HOST=db
      - PORTUS_DB_DATABASE=portus_production
      - PORTUS_DB_PASSWORD=MKFuJvPZvjhp7apE
      - PORTUS_DB_POOL=5

      # Secrets. It can possibly be handled better with Swarm's secrets.
      - PORTUS_SECRET_KEY_BASE=b4a45d52e329d70f5e05893ced9f92575a3e85b74f6a35a8596c0739ff764fda71013ea2fe885f6510d75f80a8b5f8b1f44485f8b062cefe00766e545a60e7fa
      - PORTUS_KEY_PATH=/certificates/2019_key.pem
      - PORTUS_PASSWORD=J2o3ddRVnePRLYxU

      - PORTUS_BACKGROUND=true
    links:
      - db
    volumes:
      - /srv/persistence/portus/certs:/certificates:ro
    extra_hosts:
      - "portus.xxx:192.168.252.113"
    deploy:
      placement:
        constraints: [node.hostname == srv-007]
    logging:
      driver: "gelf"
      options:
        gelf-address: "tcp://graylog.xxx:5555"
        tag: "portus-backend"
        mode: non-blocking
        max-buffer-size: 1g

  db:
    image: library/mariadb:10.0.23
    restart: unless-stopped
    command: mysqld --character-set-server=utf8 --collation-server=utf8_unicode_ci --init-connect='SET NAMES UTF8;' --innodb-flush-log-at-trx-commit=0
    environment:
      - MYSQL_DATABASE=portus_production
      - MYSQL_ROOT_PASSWORD=MKFuJvPZvjhp7apE
    volumes:
      - /srv/persistence/portus/database:/var/lib/mysql
    deploy:
      placement:
        constraints: [node.hostname == srv-007]

  registry:
    image: library/registry:2.6
    restart: unless-stopped
    command: ["/bin/sh", "/etc/docker/registry/init"]
    environment:
      # Authentication
      REGISTRY_AUTH_TOKEN_REALM: https://portus.xxxx/v2/token
      REGISTRY_AUTH_TOKEN_SERVICE: portusxxx
      REGISTRY_AUTH_TOKEN_ISSUER: portus.xxxx
      REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE: /secrets/2019_bundle.pem

      # SSL
      REGISTRY_HTTP_TLS_CERTIFICATE: /secrets/2019_cert.pem
      REGISTRY_HTTP_TLS_KEY: /secrets/2019_key.pem

      # Portus endpoint
      REGISTRY_NOTIFICATIONS_ENDPOINTS: >
        - name: portus
          url: https://portus.xxx/v2/webhooks/events
          timeout: 2000ms
          threshold: 5
          backoff: 1s
    volumes:
      - /srv/persistence/portus/registry/storage:/var/lib/registry
      - /srv/persistence/portus/certs:/secrets:ro
      - /srv/persistence/portus/registry/config/config.yml:/etc/docker/registry/config.yml:ro
      - /srv/persistence/portus/registry/init/init:/etc/docker/registry/init:ro
    ports:
      - 127.0.0.1:8064:5000
      - 127.0.0.1:8065:5001 # required to access debug service
    links:
      - portus:portus
    deploy:
      placement:
        constraints: [node.hostname == srv-007]
    logging:
      driver: "gelf"
      options:
        gelf-address: "tcp://graylog.xxx:5555"
        tag: "portus-registry"
        mode: non-blocking
        max-buffer-size: 1g

Portus version: 2.4.3

What's weird is that it used to work and, as far as I know, nothing changed.

danielbecker commented 4 years ago

After a reboot of all containers, everything seems to be working again. Not sure what the issue was.

Jean-Baptiste-Lasselle commented 4 years ago

Description

It seems like our portus installation has disconnected from the registry, even though the link icon is still green on the registry page. However, newly pushed repositories and tags are not showing up in the web interface. It is accessible through the registry. So the upload definitely worked.

Steps to reproduce

1. Push an image to the registry

2. Go to the webinterface

3. The image is not showing

In addition, I'm seeing this error message on the registry every second:

time="2019-12-04T16:06:18Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.7.6 http.request.host=portus.xxx http.request.id=4f41dc46-96f6-4f3e-8a19-869b2370674c http.request.method=GET http.request.remoteaddr=10.10.99.4 http.request.uri="/v2/deploy/xxx/manifests/sha256:cdf70ac66018503de92cea923a5b14f38e9f9362c7d6803086f18d055a923191" http.request.useragent=Ruby instance.id=1c81fe94-e4af-4642-a019-a17898ba0145 vars.name="deploy/xxx" vars.reference="sha256:cdf70ac66018503de92cea923a5b14f38e9f9362c7d6803086f18d055a923191" version=v2.6.2-14-ga66a4c3

Deployment information

The deployment is done through docker-compose with an nginx reverse proxy in front of the setup.

Configuration:

You can get this information like this:

* In bare metal execute: `bundle exec rake portus:info`.

* In a container:

  * Using the development `docker-compose.yml` file: `docker exec -it <container-id> bundle exec rake portus:info`.
  * Using the [production image](https://hub.docker.com/r/opensuse/portus/): `docker exec -it <container-id> portusctl exec rake portus:info`.
version: "3.7"

services:
  portus:
    image: opensuse/portus:2.4
    restart: unless-stopped
    environment:
      - PORTUS_MACHINE_FQDN_VALUE=portus.xxx

      # DB. The password for the database should definitely not be here. You are
      # probably better off with Docker Swarm secrets.
      - PORTUS_DB_HOST=db
      - PORTUS_DB_DATABASE=portus_production
      - PORTUS_DB_PASSWORD=MKFuJvPZvjhp7apE
      - PORTUS_DB_POOL=5

      # Secrets. It can possibly be handled better with Swarm's secrets.
      - PORTUS_SECRET_KEY_BASE=b4a45d52e329d70f5e05893ced9f92575a3e85b74f6a35a8596c0739ff764fda71013ea2fe885f6510d75f80a8b5f8b1f44485f8b062cefe00766e545a60e7fa
      - PORTUS_KEY_PATH=/certificates/2019_key.pem
      - PORTUS_PASSWORD=J2o3ddRVnePRLYxU

      # SSL
      - PORTUS_PUMA_TLS_KEY=/certificates/2019_key.pem
      - PORTUS_PUMA_TLS_CERT=/certificates/2019_cert.pem

      - RAILS_SERVE_STATIC_FILES=false
    ports:
      - 127.0.0.1:8063:3000
    links:
      - db
    volumes:
      - /srv/persistence/portus/certs:/certificates:ro
      - /srv/persistence/portus/assets:/srv/Portus/public
    extra_hosts:
      - "portus.xxx:192.168.252.113"
    deploy:
      placement:
        constraints: [node.hostname == srv-007]
    logging:
      driver: "gelf"
      options:
        gelf-address: "tcp://graylog.xxx:5555"
        tag: "portus-frontend"
        mode: non-blocking
        max-buffer-size: 1g

  background:
    image: opensuse/portus:2.4
    restart: unless-stopped
    environment:
      - CCONFIG_PREFIX=PORTUS
      - PORTUS_MACHINE_FQDN_VALUE=portus.xxx

      # DB. The password for the database should definitely not be here. You are
      # probably better off with Docker Swarm secrets.
      - PORTUS_DB_HOST=db
      - PORTUS_DB_DATABASE=portus_production
      - PORTUS_DB_PASSWORD=MKFuJvPZvjhp7apE
      - PORTUS_DB_POOL=5

      # Secrets. It can possibly be handled better with Swarm's secrets.
      - PORTUS_SECRET_KEY_BASE=b4a45d52e329d70f5e05893ced9f92575a3e85b74f6a35a8596c0739ff764fda71013ea2fe885f6510d75f80a8b5f8b1f44485f8b062cefe00766e545a60e7fa
      - PORTUS_KEY_PATH=/certificates/2019_key.pem
      - PORTUS_PASSWORD=J2o3ddRVnePRLYxU

      - PORTUS_BACKGROUND=true
    links:
      - db
    volumes:
      - /srv/persistence/portus/certs:/certificates:ro
    extra_hosts:
      - "portus.xxx:192.168.252.113"
    deploy:
      placement:
        constraints: [node.hostname == srv-007]
    logging:
      driver: "gelf"
      options:
        gelf-address: "tcp://graylog.xxx:5555"
        tag: "portus-backend"
        mode: non-blocking
        max-buffer-size: 1g

  db:
    image: library/mariadb:10.0.23
    restart: unless-stopped
    command: mysqld --character-set-server=utf8 --collation-server=utf8_unicode_ci --init-connect='SET NAMES UTF8;' --innodb-flush-log-at-trx-commit=0
    environment:
      - MYSQL_DATABASE=portus_production
      - MYSQL_ROOT_PASSWORD=MKFuJvPZvjhp7apE
    volumes:
      - /srv/persistence/portus/database:/var/lib/mysql
    deploy:
      placement:
        constraints: [node.hostname == srv-007]

  registry:
    image: library/registry:2.6
    restart: unless-stopped
    command: ["/bin/sh", "/etc/docker/registry/init"]
    environment:
      # Authentication
      REGISTRY_AUTH_TOKEN_REALM: https://portus.xxxx/v2/token
      REGISTRY_AUTH_TOKEN_SERVICE: portusxxx
      REGISTRY_AUTH_TOKEN_ISSUER: portus.xxxx
      REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE: /secrets/2019_bundle.pem

      # SSL
      REGISTRY_HTTP_TLS_CERTIFICATE: /secrets/2019_cert.pem
      REGISTRY_HTTP_TLS_KEY: /secrets/2019_key.pem

      # Portus endpoint
      REGISTRY_NOTIFICATIONS_ENDPOINTS: >
        - name: portus
          url: https://portus.xxx/v2/webhooks/events
          timeout: 2000ms
          threshold: 5
          backoff: 1s
    volumes:
      - /srv/persistence/portus/registry/storage:/var/lib/registry
      - /srv/persistence/portus/certs:/secrets:ro
      - /srv/persistence/portus/registry/config/config.yml:/etc/docker/registry/config.yml:ro
      - /srv/persistence/portus/registry/init/init:/etc/docker/registry/init:ro
    ports:
      - 127.0.0.1:8064:5000
      - 127.0.0.1:8065:5001 # required to access debug service
    links:
      - portus:portus
    deploy:
      placement:
        constraints: [node.hostname == srv-007]
    logging:
      driver: "gelf"
      options:
        gelf-address: "tcp://graylog.xxx:5555"
        tag: "portus-registry"
        mode: non-blocking
        max-buffer-size: 1g

Portus version: 2.4.3

What's weird is that it used to work and, as far as I know, nothing changed.

Hi @danielbecker This often happens because of a problem related to the webhooks sent from registry, to portuys, as notifications of registry events (like docker push/pulls)