Closed danielbecker closed 4 years ago
After a reboot of all containers, everything seems to be working again. Not sure what the issue was.
Description
It seems like our portus installation has disconnected from the registry, even though the link icon is still green on the registry page. However, newly pushed repositories and tags are not showing up in the web interface. It is accessible through the registry. So the upload definitely worked.
Steps to reproduce
1. Push an image to the registry 2. Go to the webinterface 3. The image is not showing
In addition, I'm seeing this error message on the registry every second:
time="2019-12-04T16:06:18Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.7.6 http.request.host=portus.xxx http.request.id=4f41dc46-96f6-4f3e-8a19-869b2370674c http.request.method=GET http.request.remoteaddr=10.10.99.4 http.request.uri="/v2/deploy/xxx/manifests/sha256:cdf70ac66018503de92cea923a5b14f38e9f9362c7d6803086f18d055a923191" http.request.useragent=Ruby instance.id=1c81fe94-e4af-4642-a019-a17898ba0145 vars.name="deploy/xxx" vars.reference="sha256:cdf70ac66018503de92cea923a5b14f38e9f9362c7d6803086f18d055a923191" version=v2.6.2-14-ga66a4c3
Deployment information
The deployment is done through docker-compose with an nginx reverse proxy in front of the setup.
Configuration:
You can get this information like this:
* In bare metal execute: `bundle exec rake portus:info`. * In a container: * Using the development `docker-compose.yml` file: `docker exec -it <container-id> bundle exec rake portus:info`. * Using the [production image](https://hub.docker.com/r/opensuse/portus/): `docker exec -it <container-id> portusctl exec rake portus:info`.
version: "3.7" services: portus: image: opensuse/portus:2.4 restart: unless-stopped environment: - PORTUS_MACHINE_FQDN_VALUE=portus.xxx # DB. The password for the database should definitely not be here. You are # probably better off with Docker Swarm secrets. - PORTUS_DB_HOST=db - PORTUS_DB_DATABASE=portus_production - PORTUS_DB_PASSWORD=MKFuJvPZvjhp7apE - PORTUS_DB_POOL=5 # Secrets. It can possibly be handled better with Swarm's secrets. - PORTUS_SECRET_KEY_BASE=b4a45d52e329d70f5e05893ced9f92575a3e85b74f6a35a8596c0739ff764fda71013ea2fe885f6510d75f80a8b5f8b1f44485f8b062cefe00766e545a60e7fa - PORTUS_KEY_PATH=/certificates/2019_key.pem - PORTUS_PASSWORD=J2o3ddRVnePRLYxU # SSL - PORTUS_PUMA_TLS_KEY=/certificates/2019_key.pem - PORTUS_PUMA_TLS_CERT=/certificates/2019_cert.pem - RAILS_SERVE_STATIC_FILES=false ports: - 127.0.0.1:8063:3000 links: - db volumes: - /srv/persistence/portus/certs:/certificates:ro - /srv/persistence/portus/assets:/srv/Portus/public extra_hosts: - "portus.xxx:192.168.252.113" deploy: placement: constraints: [node.hostname == srv-007] logging: driver: "gelf" options: gelf-address: "tcp://graylog.xxx:5555" tag: "portus-frontend" mode: non-blocking max-buffer-size: 1g background: image: opensuse/portus:2.4 restart: unless-stopped environment: - CCONFIG_PREFIX=PORTUS - PORTUS_MACHINE_FQDN_VALUE=portus.xxx # DB. The password for the database should definitely not be here. You are # probably better off with Docker Swarm secrets. - PORTUS_DB_HOST=db - PORTUS_DB_DATABASE=portus_production - PORTUS_DB_PASSWORD=MKFuJvPZvjhp7apE - PORTUS_DB_POOL=5 # Secrets. It can possibly be handled better with Swarm's secrets. - PORTUS_SECRET_KEY_BASE=b4a45d52e329d70f5e05893ced9f92575a3e85b74f6a35a8596c0739ff764fda71013ea2fe885f6510d75f80a8b5f8b1f44485f8b062cefe00766e545a60e7fa - PORTUS_KEY_PATH=/certificates/2019_key.pem - PORTUS_PASSWORD=J2o3ddRVnePRLYxU - PORTUS_BACKGROUND=true links: - db volumes: - /srv/persistence/portus/certs:/certificates:ro extra_hosts: - "portus.xxx:192.168.252.113" deploy: placement: constraints: [node.hostname == srv-007] logging: driver: "gelf" options: gelf-address: "tcp://graylog.xxx:5555" tag: "portus-backend" mode: non-blocking max-buffer-size: 1g db: image: library/mariadb:10.0.23 restart: unless-stopped command: mysqld --character-set-server=utf8 --collation-server=utf8_unicode_ci --init-connect='SET NAMES UTF8;' --innodb-flush-log-at-trx-commit=0 environment: - MYSQL_DATABASE=portus_production - MYSQL_ROOT_PASSWORD=MKFuJvPZvjhp7apE volumes: - /srv/persistence/portus/database:/var/lib/mysql deploy: placement: constraints: [node.hostname == srv-007] registry: image: library/registry:2.6 restart: unless-stopped command: ["/bin/sh", "/etc/docker/registry/init"] environment: # Authentication REGISTRY_AUTH_TOKEN_REALM: https://portus.xxxx/v2/token REGISTRY_AUTH_TOKEN_SERVICE: portusxxx REGISTRY_AUTH_TOKEN_ISSUER: portus.xxxx REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE: /secrets/2019_bundle.pem # SSL REGISTRY_HTTP_TLS_CERTIFICATE: /secrets/2019_cert.pem REGISTRY_HTTP_TLS_KEY: /secrets/2019_key.pem # Portus endpoint REGISTRY_NOTIFICATIONS_ENDPOINTS: > - name: portus url: https://portus.xxx/v2/webhooks/events timeout: 2000ms threshold: 5 backoff: 1s volumes: - /srv/persistence/portus/registry/storage:/var/lib/registry - /srv/persistence/portus/certs:/secrets:ro - /srv/persistence/portus/registry/config/config.yml:/etc/docker/registry/config.yml:ro - /srv/persistence/portus/registry/init/init:/etc/docker/registry/init:ro ports: - 127.0.0.1:8064:5000 - 127.0.0.1:8065:5001 # required to access debug service links: - portus:portus deploy: placement: constraints: [node.hostname == srv-007] logging: driver: "gelf" options: gelf-address: "tcp://graylog.xxx:5555" tag: "portus-registry" mode: non-blocking max-buffer-size: 1g
Portus version: 2.4.3
What's weird is that it used to work and, as far as I know, nothing changed.
Hi @danielbecker This often happens because of a problem related to the webhooks sent from registry, to portuys, as notifications of registry events (like docker push/pulls)
Description
It seems like our portus installation has disconnected from the registry, even though the link icon is still green on the registry page. However, newly pushed repositories and tags are not showing up in the web interface. It is accessible through the registry. So the upload definitely worked.
Steps to reproduce
In addition, I'm seeing this error message on the registry every second:
time="2019-12-04T16:06:18Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.7.6 http.request.host=portus.xxx http.request.id=4f41dc46-96f6-4f3e-8a19-869b2370674c http.request.method=GET http.request.remoteaddr=10.10.99.4 http.request.uri="/v2/deploy/xxx/manifests/sha256:cdf70ac66018503de92cea923a5b14f38e9f9362c7d6803086f18d055a923191" http.request.useragent=Ruby instance.id=1c81fe94-e4af-4642-a019-a17898ba0145 vars.name="deploy/xxx" vars.reference="sha256:cdf70ac66018503de92cea923a5b14f38e9f9362c7d6803086f18d055a923191" version=v2.6.2-14-ga66a4c3
Deployment information
The deployment is done through docker-compose with an nginx reverse proxy in front of the setup.
Configuration:
You can get this information like this:
bundle exec rake portus:info
.docker-compose.yml
file:docker exec -it <container-id> bundle exec rake portus:info
.docker exec -it <container-id> portusctl exec rake portus:info
.Portus version: 2.4.3
What's weird is that it used to work and, as far as I know, nothing changed.