Closed kinland closed 4 years ago
Additional observation: upon restarting Portus, there are now five docker images running (as opposed to 4). There are two containers running opensuse/portus:head
with /init
command:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b7bb7557bcf4 nginx:alpine "nginx -g 'daemon of…" 6 weeks ago Up 3 weeks 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp portus_nginx_1
5035ec47f25b opensuse/portus:head "/init" 6 weeks ago Up 3 weeks 3000/tcp portus_background_1
d9d7f7f8ca0f registry:2.6 "/entrypoint.sh /bin…" 6 weeks ago Up 3 weeks 0.0.0.0:5000-5001->5000-5001/tcp portus_registry_1
337b399fcd28 mariadb:10.0.23 "/docker-entrypoint.…" 6 weeks ago Up 3 weeks 3306/tcp portus_db_1
vs
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b7bb7557bcf4 nginx:alpine "nginx -g 'daemon of…" 6 weeks ago Up 17 seconds 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp portus_nginx_1
5035ec47f25b opensuse/portus:head "/init" 6 weeks ago Up 15 seconds 3000/tcp portus_background_1
d9d7f7f8ca0f registry:2.6 "/entrypoint.sh /bin…" 6 weeks ago Up 6 seconds 0.0.0.0:5000-5001->5000-5001/tcp portus_registry_1
e93a32f5e405 opensuse/portus:head "/init" 6 weeks ago Up 6 seconds 0.0.0.0:3000->3000/tcp portus_portus_1
337b399fcd28 mariadb:10.0.23 "/docker-entrypoint.…" 6 weeks ago Up 15 seconds 3306/tcp portus_db_1
I noticed this in the logs for that container that appeared upon restart. Maybe it's a memory leak? (Cannot allocate memory - fork(2)
)
Started POST "/v2/webhooks/events" for 172.21.0.6 at 2019-11-05 22:28:10 +0000
Processing by Api::V2::EventsController#create as JSON
Completed 202 Accepted in 7ms (ActiveRecord: 4.2ms)
[31] ! Terminating timed out worker: 174
Cannot allocate memory - fork(2)
[151] ! Detected parent died, dying
[152] ! Detected parent died, dying
/srv/Portus/vendor/bundle/ruby/2.6.0/gems/puma-3.12.1/lib/puma/cluster.rb:139:in `fork'
/srv/Portus/vendor/bundle/ruby/2.6.0/gems/puma-3.12.1/lib/puma/cluster.rb:139:in `block in spawn_workers'
/srv/Portus/vendor/bundle/ruby/2.6.0/gems/puma-3.12.1/lib/puma/cluster.rb:135:in `times'
/srv/Portus/vendor/bundle/ruby/2.6.0/gems/puma-3.12.1/lib/puma/cluster.rb:135:in `spawn_workers'
/srv/Portus/vendor/bundle/ruby/2.6.0/gems/puma-3.12.1/lib/puma/cluster.rb:213:in `check_workers'
/srv/Portus/vendor/bundle/ruby/2.6.0/gems/puma-3.12.1/lib/puma/cluster.rb:486:in `run'
/srv/Portus/vendor/bundle/ruby/2.6.0/gems/puma-3.12.1/lib/puma/launcher.rb:186:in `run'
/srv/Portus/vendor/bundle/ruby/2.6.0/gems/puma-3.12.1/lib/puma/cli.rb:80:in `run'
/srv/Portus/vendor/bundle/ruby/2.6.0/gems/puma-3.12.1/lib/puma/control_cli.rb:261:in `start'
/srv/Portus/vendor/bundle/ruby/2.6.0/gems/puma-3.12.1/lib/puma/control_cli.rb:225:in `run'
/srv/Portus/vendor/bundle/ruby/2.6.0/gems/puma-3.12.1/bin/pumactl:8:in `<top (required)>'
/srv/Portus/vendor/bundle/ruby/2.6.0/bin/pumactl:23:in `load'
/srv/Portus/vendor/bundle/ruby/2.6.0/bin/pumactl:23:in `<main>'
[84] ! Detected parent died, dying
exit status 1
/usr/bin/bundle:23:in `load': cannot load such file -- /usr/lib64/ruby/gems/2.6.0/gems/bundler-1.16.4/exe/bundle (LoadError)
from /usr/bin/bundle:23:in `<main>'
portus_portus_1 comes from the docker_compose.yml:
ubuntu@<ip-aws-ec2>:~/portus$ cat docker-compose.yml
version: "2"
services:
portus:
image: opensuse/portus:head
environment:
- PORTUS_MACHINE_FQDN_VALUE=${MACHINE_FQDN}
# DB. The password for the database should definitely not be here. You are
# probably better off with Docker Swarm secrets.
- PORTUS_DB_HOST=db
- PORTUS_DB_DATABASE=portus_production
- PORTUS_DB_PASSWORD=${DATABASE_PASSWORD}
- PORTUS_DB_POOL=5
# Secrets. It can possibly be handled better with Swarm's secrets.
- PORTUS_SECRET_KEY_BASE=${SECRET_KEY_BASE}
- PORTUS_KEY_PATH=/certificates/portus.key
- PORTUS_PASSWORD=${PORTUS_PASSWORD}
# SSL
- PORTUS_PUMA_TLS_KEY=/certificates/portus.key
- PORTUS_PUMA_TLS_CERT=/certificates/portus.crt
# NGinx is serving the assets instead of Puma. If you want to change this,
# uncomment this line.
#- RAILS_SERVE_STATIC_FILES='true'
ports:
- 3000:3000
links:
- db
volumes:
- ./secrets:/certificates:ro
- static:/srv/Portus/public
- ./config-github.yml:/srv/Portus/config/config-local.yml:ro
background:
image: opensuse/portus:head
depends_on:
- portus
- db
environment:
# Theoretically not needed, but cconfig's been buggy on this...
- CCONFIG_PREFIX=PORTUS
- PORTUS_MACHINE_FQDN_VALUE=${MACHINE_FQDN}
# DB. The password for the database should definitely not be here. You are
# probably better off with Docker Swarm secrets.
- PORTUS_DB_HOST=db
- PORTUS_DB_DATABASE=portus_production
- PORTUS_DB_PASSWORD=${DATABASE_PASSWORD}
- PORTUS_DB_POOL=5
# Secrets. It can possibly be handled better with Swarm's secrets.
- PORTUS_SECRET_KEY_BASE=${SECRET_KEY_BASE}
- PORTUS_KEY_PATH=/certificates/portus.key
- PORTUS_PASSWORD=${PORTUS_PASSWORD}
- PORTUS_BACKGROUND=true
links:
- db
volumes:
- ./secrets:/certificates:ro
db:
image: library/mariadb:10.0.23
command: mysqld --character-set-server=utf8 --collation-server=utf8_unicode_ci --init-connect='SET NAMES UTF8;' --innodb-flush-log-at-trx-commit=0
environment:
- MYSQL_DATABASE=portus_production
# Again, the password shouldn't be handled like this.
- MYSQL_ROOT_PASSWORD=${DATABASE_PASSWORD}
volumes:
- ./portus/mariadb:/var/lib/mysql
registry:
image: library/registry:2.6
command: ["/bin/sh", "/etc/docker/registry/init"]
environment:
# Authentication
REGISTRY_AUTH_TOKEN_REALM: https://${MACHINE_FQDN}/v2/token
REGISTRY_AUTH_TOKEN_SERVICE: ${MACHINE_FQDN}
REGISTRY_AUTH_TOKEN_ISSUER: ${MACHINE_FQDN}
REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE: /secrets/portus.crt
# SSL
REGISTRY_HTTP_TLS_CERTIFICATE: /secrets/portus.crt
REGISTRY_HTTP_TLS_KEY: /secrets/portus.key
# Portus endpoint
REGISTRY_NOTIFICATIONS_ENDPOINTS: >
- name: portus
url: https://${MACHINE_FQDN}/v2/webhooks/events
timeout: 2000ms
threshold: 5
backoff: 1s
volumes:
- ./portus/registry:/var/lib/registry
- ./secrets:/secrets:ro
- ./registry/config.yml:/etc/docker/registry/config.yml:ro
- ./registry/init:/etc/docker/registry/init:ro
ports:
- 5000:5000
- 5001:5001 # required to access debug service
links:
- portus:portus
nginx:
image: library/nginx:alpine
volumes:
- ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
- ./secrets:/secrets:ro
- static:/srv/Portus/public:ro
ports:
- 80:80
- 443:443
links:
- registry:registry
- portus:portus
volumes:
static:
driver: local
Hi @dhorkin I read in details all your infos, and :
docker-compose up -d --force-recreate
# don't add the [--build] option
# and before you force recreate, you have to check first that
# you've got enough RAM available on
# the containerization host, where you execute
# docker commands
Thanks for all your contributions! This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Description
After our Portus docker image has been up for ~1 month, we start getting
502 Bad Gateway
errors when we try to access Portus via a web browser.If we run
docker logs <portus_container_id>
, the last event is a pull request for an image... no errors.We're unfortunately unable to set
PORTUS_LOG_LEVEL
todebug
because Docker won't let you change an environment variable on a running container (quite an oversight). We're going to do that when we restart the container, so the next time this occurs, we'll have more information.We did, however, find that we could get an error to show up using portusctrl...
Steps to reproduce
Unfortunately the person who set up our Portus install has left the company, but the
workaround
that he provided was just restart it (docker-compose restart
).docker exec <container> portusctl -u=<user> -t=<token> -s=0.0.0.0:3000 health
Output of our nginx container logs looks like this:
Deployment information
Deployment method: We have Portus hosted on an AWS EC2 instance. Output of
docker ps
looks like so:Configuration:
It's odd that github would say it's disabled, as we're using github authentication to log into Portus.