kinland
commented
4 years ago Additional observation: upon restarting Portus, there are now five docker images running (as opposed to 4). There are two containers running opensuse/portus:head with /init command:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b7bb7557bcf4 nginx:alpine "nginx -g 'daemon of…" 6 weeks ago Up 3 weeks 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp portus_nginx_1
5035ec47f25b opensuse/portus:head "/init" 6 weeks ago Up 3 weeks 3000/tcp portus_background_1
d9d7f7f8ca0f registry:2.6 "/entrypoint.sh /bin…" 6 weeks ago Up 3 weeks 0.0.0.0:5000-5001->5000-5001/tcp portus_registry_1
337b399fcd28 mariadb:10.0.23 "/docker-entrypoint.…" 6 weeks ago Up 3 weeks 3306/tcp portus_db_1vs
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b7bb7557bcf4 nginx:alpine "nginx -g 'daemon of…" 6 weeks ago Up 17 seconds 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp portus_nginx_1
5035ec47f25b opensuse/portus:head "/init" 6 weeks ago Up 15 seconds 3000/tcp portus_background_1
d9d7f7f8ca0f registry:2.6 "/entrypoint.sh /bin…" 6 weeks ago Up 6 seconds 0.0.0.0:5000-5001->5000-5001/tcp portus_registry_1
e93a32f5e405 opensuse/portus:head "/init" 6 weeks ago Up 6 seconds 0.0.0.0:3000->3000/tcp portus_portus_1
337b399fcd28 mariadb:10.0.23 "/docker-entrypoint.…" 6 weeks ago Up 15 seconds 3306/tcp portus_db_1I noticed this in the logs for that container that appeared upon restart. Maybe it's a memory leak? (Cannot allocate memory - fork(2))
Started POST "/v2/webhooks/events" for 172.21.0.6 at 2019-11-05 22:28:10 +0000
Processing by Api::V2::EventsController#create as JSON
Completed 202 Accepted in 7ms (ActiveRecord: 4.2ms)
[31] ! Terminating timed out worker: 174
Cannot allocate memory - fork(2)
[151] ! Detected parent died, dying
[152] ! Detected parent died, dying
/srv/Portus/vendor/bundle/ruby/2.6.0/gems/puma-3.12.1/lib/puma/cluster.rb:139:in `fork'
/srv/Portus/vendor/bundle/ruby/2.6.0/gems/puma-3.12.1/lib/puma/cluster.rb:139:in `block in spawn_workers'
/srv/Portus/vendor/bundle/ruby/2.6.0/gems/puma-3.12.1/lib/puma/cluster.rb:135:in `times'
/srv/Portus/vendor/bundle/ruby/2.6.0/gems/puma-3.12.1/lib/puma/cluster.rb:135:in `spawn_workers'
/srv/Portus/vendor/bundle/ruby/2.6.0/gems/puma-3.12.1/lib/puma/cluster.rb:213:in `check_workers'
/srv/Portus/vendor/bundle/ruby/2.6.0/gems/puma-3.12.1/lib/puma/cluster.rb:486:in `run'
/srv/Portus/vendor/bundle/ruby/2.6.0/gems/puma-3.12.1/lib/puma/launcher.rb:186:in `run'
/srv/Portus/vendor/bundle/ruby/2.6.0/gems/puma-3.12.1/lib/puma/cli.rb:80:in `run'
/srv/Portus/vendor/bundle/ruby/2.6.0/gems/puma-3.12.1/lib/puma/control_cli.rb:261:in `start'
/srv/Portus/vendor/bundle/ruby/2.6.0/gems/puma-3.12.1/lib/puma/control_cli.rb:225:in `run'
/srv/Portus/vendor/bundle/ruby/2.6.0/gems/puma-3.12.1/bin/pumactl:8:in `<top (required)>'
/srv/Portus/vendor/bundle/ruby/2.6.0/bin/pumactl:23:in `load'
/srv/Portus/vendor/bundle/ruby/2.6.0/bin/pumactl:23:in `<main>'
[84] ! Detected parent died, dying
exit status 1
/usr/bin/bundle:23:in `load': cannot load such file -- /usr/lib64/ruby/gems/2.6.0/gems/bundler-1.16.4/exe/bundle (LoadError)
from /usr/bin/bundle:23:in `<main>'portus_portus_1 comes from the docker_compose.yml:
ubuntu@<ip-aws-ec2>:~/portus$ cat docker-compose.yml
version: "2"
services:
portus:
image: opensuse/portus:head
environment:
- PORTUS_MACHINE_FQDN_VALUE=${MACHINE_FQDN}
# DB. The password for the database should definitely not be here. You are
# probably better off with Docker Swarm secrets.
- PORTUS_DB_HOST=db
- PORTUS_DB_DATABASE=portus_production
- PORTUS_DB_PASSWORD=${DATABASE_PASSWORD}
- PORTUS_DB_POOL=5
# Secrets. It can possibly be handled better with Swarm's secrets.
- PORTUS_SECRET_KEY_BASE=${SECRET_KEY_BASE}
- PORTUS_KEY_PATH=/certificates/portus.key
- PORTUS_PASSWORD=${PORTUS_PASSWORD}
# SSL
- PORTUS_PUMA_TLS_KEY=/certificates/portus.key
- PORTUS_PUMA_TLS_CERT=/certificates/portus.crt
# NGinx is serving the assets instead of Puma. If you want to change this,
# uncomment this line.
#- RAILS_SERVE_STATIC_FILES='true'
ports:
- 3000:3000
links:
- db
volumes:
- ./secrets:/certificates:ro
- static:/srv/Portus/public
- ./config-github.yml:/srv/Portus/config/config-local.yml:ro
background:
image: opensuse/portus:head
depends_on:
- portus
- db
environment:
# Theoretically not needed, but cconfig's been buggy on this...
- CCONFIG_PREFIX=PORTUS
- PORTUS_MACHINE_FQDN_VALUE=${MACHINE_FQDN}
# DB. The password for the database should definitely not be here. You are
# probably better off with Docker Swarm secrets.
- PORTUS_DB_HOST=db
- PORTUS_DB_DATABASE=portus_production
- PORTUS_DB_PASSWORD=${DATABASE_PASSWORD}
- PORTUS_DB_POOL=5
# Secrets. It can possibly be handled better with Swarm's secrets.
- PORTUS_SECRET_KEY_BASE=${SECRET_KEY_BASE}
- PORTUS_KEY_PATH=/certificates/portus.key
- PORTUS_PASSWORD=${PORTUS_PASSWORD}
- PORTUS_BACKGROUND=true
links:
- db
volumes:
- ./secrets:/certificates:ro
db:
image: library/mariadb:10.0.23
command: mysqld --character-set-server=utf8 --collation-server=utf8_unicode_ci --init-connect='SET NAMES UTF8;' --innodb-flush-log-at-trx-commit=0
environment:
- MYSQL_DATABASE=portus_production
# Again, the password shouldn't be handled like this.
- MYSQL_ROOT_PASSWORD=${DATABASE_PASSWORD}
volumes:
- ./portus/mariadb:/var/lib/mysql
registry:
image: library/registry:2.6
command: ["/bin/sh", "/etc/docker/registry/init"]
environment:
# Authentication
REGISTRY_AUTH_TOKEN_REALM: https://${MACHINE_FQDN}/v2/token
REGISTRY_AUTH_TOKEN_SERVICE: ${MACHINE_FQDN}
REGISTRY_AUTH_TOKEN_ISSUER: ${MACHINE_FQDN}
REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE: /secrets/portus.crt
# SSL
REGISTRY_HTTP_TLS_CERTIFICATE: /secrets/portus.crt
REGISTRY_HTTP_TLS_KEY: /secrets/portus.key
# Portus endpoint
REGISTRY_NOTIFICATIONS_ENDPOINTS: >
- name: portus
url: https://${MACHINE_FQDN}/v2/webhooks/events
timeout: 2000ms
threshold: 5
backoff: 1s
volumes:
- ./portus/registry:/var/lib/registry
- ./secrets:/secrets:ro
- ./registry/config.yml:/etc/docker/registry/config.yml:ro
- ./registry/init:/etc/docker/registry/init:ro
ports:
- 5000:5000
- 5001:5001 # required to access debug service
links:
- portus:portus
nginx:
image: library/nginx:alpine
volumes:
- ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
- ./secrets:/secrets:ro
- static:/srv/Portus/public:ro
ports:
- 80:80
- 443:443
links:
- registry:registry
- portus:portus
volumes:
static:
driver: local
Jean-Baptiste-Lasselle
stale[bot]
Description
After our Portus docker image has been up for ~1 month, we start getting
502 Bad Gatewayerrors when we try to access Portus via a web browser.If we run
docker logs <portus_container_id>, the last event is a pull request for an image... no errors.We're unfortunately unable to set
PORTUS_LOG_LEVELtodebugbecause Docker won't let you change an environment variable on a running container (quite an oversight). We're going to do that when we restart the container, so the next time this occurs, we'll have more information.We did, however, find that we could get an error to show up using portusctrl...
Steps to reproduce
Unfortunately the person who set up our Portus install has left the company, but the
workaroundthat he provided was just restart it (docker-compose restart).docker exec <container> portusctl -u=<user> -t=<token> -s=0.0.0.0:3000 healthOutput of our nginx container logs looks like this:
Deployment information
Deployment method: We have Portus hosted on an AWS EC2 instance. Output of
docker pslooks like so:Configuration:
It's odd that github would say it's disabled, as we're using github authentication to log into Portus.