search

sameersbn / docker-gitlab

Dockerized GitLab
http://www.damagehead.com/docker-gitlab/
MIT License
7.89k stars 2.14k forks source link

exited: unicorn (exit status 1; not expected) #1861

Closed milovidov983 closed 5 years ago

milovidov983 commented 5 years ago

I try migrate old version sameersbn/docker-gitlab to new. I copied all the gitlab files to a other directory and hooked them into the docker-compose file:

version: '2'

services:
  redis:
    restart: always
    image: sameersbn/redis:4.0.9-1
    container_name: gitlabredis4
    command:
    - --loglevel warning
    volumes:
    - /c/temp/gitlab2/git11.7/redis:/var/lib/redis:Z

  postgresql:
    restart: always
    image: sameersbn/postgresql:10
    container_name: gitlabpostgresql10
    volumes:
    - /c/temp/gitlab2/git11.7/postgresql/data:/var/lib/postgresql:Z
    environment:
    - DB_USER=gitlab
    - DB_PASS=password
    - DB_NAME=gitlabhq_production
    - DB_EXTENSION=pg_trgm

  gitlab:
    restart: always
    image: sameersbn/gitlab:11.7.5
    container_name: gitlab11
    depends_on:
    - redis
    - postgresql
    ports:
    - "10080:80"
    - "10022:22"
    volumes:
    - /c/temp/gitlab2/git11.7/data:/home/git/data:Z
    environment:
    - DEBUG=false

    - DB_ADAPTER=postgresql
    - DB_HOST=postgresql
    - DB_PORT=5432
    - DB_USER=gitlab
    - DB_PASS=password
    - DB_NAME=gitlabhq_production

    - REDIS_HOST=redis
    - REDIS_PORT=6379

    - TZ=Europe/Moscow
    - GITLAB_TIMEZONE=Moscow

    - GITLAB_HTTPS=false
    - SSL_SELF_SIGNED=false

    - GITLAB_HOST=***
    - GITLAB_PORT=10080
    - GITLAB_SSH_PORT=10022
    - GITLAB_RELATIVE_URL_ROOT=***
    - GITLAB_SECRETS_DB_KEY_BASE=***
    - GITLAB_SECRETS_SECRET_KEY_BASE=***
    - GITLAB_SECRETS_OTP_KEY_BASE=***

    - GITLAB_ROOT_PASSWORD=***
    - GITLAB_ROOT_EMAIL=***

    - GITLAB_NOTIFY_ON_BROKEN_BUILDS=true
    - GITLAB_NOTIFY_PUSHER=false

    - GITLAB_EMAIL=***
    - GITLAB_EMAIL_REPLY_TO=***
    - GITLAB_INCOMING_EMAIL_ADDRESS=***

    - GITLAB_BACKUP_SCHEDULE=daily
    - GITLAB_BACKUP_TIME=01:00

    - SMTP_ENABLED=false
    - SMTP_DOMAIN=www.example.com
    - SMTP_HOST=***
    - SMTP_PORT=587
    - SMTP_USER=***
    - SMTP_PASS=***
    - SMTP_STARTTLS=true
    - SMTP_AUTHENTICATION=login

    - IMAP_ENABLED=false
    - OAUTH_ENABLED=false

That's what gitlab writes in the logs:


Loading /etc/docker-gitlab/runtime/env-defaults
Initializing logdir...
Initializing datadir...
Installing configuration templates...
Configuring gitlab...
Configuring gitlab::database
Configuring gitlab::redis
Configuring gitlab::secrets...
Configuring gitlab::sidekiq...
Configuring gitlab::gitaly...
Configuring gitlab::monitoring...
Configuring gitlab::gitlab-workhorse...
Configuring gitlab::relative_url...
Configuring gitlab::unicorn...
Configuring gitlab::timezone...
Configuring gitlab::rack_attack...
Configuring gitlab::ci...
Configuring gitlab::artifacts...
Configuring gitlab::lfs...
Configuring gitlab::uploads...
Configuring gitlab::mattermost...
Configuring gitlab::project_features...
Configuring gitlab::oauth...
Configuring gitlab::ldap...
Configuring gitlab::cron_jobs...
Configuring gitlab::backups...
Configuring gitlab::backups::schedule...
Configuring gitlab::registry...
Configuring gitlab::pages...
Configuring gitlab-shell...
Configuring nginx...
Configuring nginx::gitlab...
Setting up GitLab for firstrun. Please be patient, this could take a while...
2019-02-13 17:19:04,847 CRIT Supervisor running as root (no user in config file)
2019-02-13 17:19:04,847 WARN Included extra file "/etc/supervisor/conf.d/cron.conf" during parsing
2019-02-13 17:19:04,847 WARN Included extra file "/etc/supervisor/conf.d/gitaly.conf" during parsing
2019-02-13 17:19:04,848 WARN Included extra file "/etc/supervisor/conf.d/gitlab-workhorse.conf" during parsing
2019-02-13 17:19:04,848 WARN Included extra file "/etc/supervisor/conf.d/mail_room.conf" during parsing
2019-02-13 17:19:04,848 WARN Included extra file "/etc/supervisor/conf.d/nginx.conf" during parsing
2019-02-13 17:19:04,848 WARN Included extra file "/etc/supervisor/conf.d/sidekiq.conf" during parsing
2019-02-13 17:19:04,848 WARN Included extra file "/etc/supervisor/conf.d/sshd.conf" during parsing
2019-02-13 17:19:04,848 WARN Included extra file "/etc/supervisor/conf.d/unicorn.conf" during parsing
2019-02-13 17:19:05,115 INFO RPC interface 'supervisor' initialized
2019-02-13 17:19:05,115 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2019-02-13 17:19:05,116 INFO supervisord started with pid 602
2019-02-13 17:19:06,190 INFO spawned: 'gitaly' with pid 618
2019-02-13 17:19:06,193 INFO spawned: 'sidekiq' with pid 619
2019-02-13 17:19:06,196 INFO spawned: 'unicorn' with pid 620
2019-02-13 17:19:06,199 INFO spawned: 'gitlab-workhorse' with pid 621
2019-02-13 17:19:06,202 INFO spawned: 'cron' with pid 622
2019-02-13 17:19:06,210 INFO spawned: 'nginx' with pid 624
2019-02-13 17:19:06,228 INFO spawned: 'sshd' with pid 625
2019-02-13 17:19:07,230 INFO success: gitaly entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-02-13 17:19:07,230 INFO success: sidekiq entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-02-13 17:19:07,230 INFO success: unicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-02-13 17:19:07,231 INFO success: gitlab-workhorse entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-02-13 17:19:07,231 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-02-13 17:19:07,231 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-02-13 17:19:07,231 INFO success: sshd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Database 'gitlabhq_production' already exists
2019-02-13 17:20:48,383 INFO exited: unicorn (exit status 1; not expected)
2019-02-13 17:20:49,438 INFO spawned: 'unicorn' with pid 818
2019-02-13 17:20:51,273 INFO success: unicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-02-13 17:21:17,781 INFO exited: unicorn (exit status 1; not expected)
2019-02-13 17:21:18,784 INFO spawned: 'unicorn' with pid 856
2019-02-13 17:21:20,459 INFO success: unicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-02-13 17:21:27,223 WARN received SIGTERM indicating exit request
2019-02-13 17:21:27,223 INFO waiting for sshd, gitlab-workhorse, sidekiq, cron, nginx, gitaly, unicorn to die
2019-02-13 17:21:27,235 INFO stopped: sshd (exit status 0)
2019-02-13 17:21:27,276 INFO stopped: nginx (exit status 0)
2019-02-13 17:21:27,277 INFO stopped: cron (terminated by SIGTERM)
2019-02-13 17:21:27,279 INFO stopped: gitlab-workhorse (terminated by SIGTERM)
2019-02-13 17:21:30,283 INFO waiting for sidekiq, gitaly, unicorn to die
2019-02-13 17:21:33,287 INFO waiting for sidekiq, gitaly, unicorn to die
2019-02-13 17:21:36,291 INFO waiting for sidekiq, gitaly, unicorn to die
2019-02-13 17:21:38,294 WARN killing 'unicorn' (856) with SIGKILL
2019-02-13 17:21:38,311 INFO stopped: unicorn (terminated by SIGKILL)
2019-02-13 17:21:39,312 INFO waiting for sidekiq, gitaly to die
2019-02-13 17:21:40,971 INFO stopped: sidekiq (exit status 0)
2019-02-13 17:21:41,001 INFO stopped: gitaly (exit status 1)
2019-02-13 17:21:41,506 CRIT Supervisor running as root (no user in config file)
2019-02-13 17:21:41,506 WARN Included extra file "/etc/supervisor/conf.d/cron.conf" during parsing
2019-02-13 17:21:41,506 WARN Included extra file "/etc/supervisor/conf.d/gitaly.conf" during parsing
2019-02-13 17:21:41,506 WARN Included extra file "/etc/supervisor/conf.d/gitlab-workhorse.conf" during parsing
2019-02-13 17:21:41,507 WARN Included extra file "/etc/supervisor/conf.d/mail_room.conf" during parsing
2019-02-13 17:21:41,507 WARN Included extra file "/etc/supervisor/conf.d/nginx.conf" during parsing
2019-02-13 17:21:41,507 WARN Included extra file "/etc/supervisor/conf.d/sidekiq.conf" during parsing
2019-02-13 17:21:41,507 WARN Included extra file "/etc/supervisor/conf.d/sshd.conf" during parsing
2019-02-13 17:21:41,507 WARN Included extra file "/etc/supervisor/conf.d/unicorn.conf" during parsing
2019-02-13 17:21:41,517 INFO RPC interface 'supervisor' initialized
2019-02-13 17:21:41,517 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2019-02-13 17:21:41,518 INFO supervisord started with pid 1
2019-02-13 17:21:42,520 INFO spawned: 'gitaly' with pid 869
2019-02-13 17:21:42,522 INFO spawned: 'sidekiq' with pid 870
2019-02-13 17:21:42,525 INFO spawned: 'unicorn' with pid 871
2019-02-13 17:21:42,527 INFO spawned: 'gitlab-workhorse' with pid 872
2019-02-13 17:21:42,537 INFO spawned: 'cron' with pid 873
2019-02-13 17:21:42,549 INFO spawned: 'nginx' with pid 879
2019-02-13 17:21:42,563 INFO spawned: 'sshd' with pid 883
2019-02-13 17:21:44,033 INFO success: gitaly entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-02-13 17:21:44,033 INFO success: sidekiq entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-02-13 17:21:44,033 INFO success: unicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-02-13 17:21:44,033 INFO success: gitlab-workhorse entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-02-13 17:21:44,033 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-02-13 17:21:44,033 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-02-13 17:21:44,033 INFO success: sshd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-02-13 17:22:22,080 INFO exited: unicorn (exit status 1; not expected)
2019-02-13 17:22:23,083 INFO spawned: 'unicorn' with pid 1029
2019-02-13 17:22:24,713 INFO success: unicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-02-13 17:22:48,794 INFO exited: unicorn (exit status 1; not expected)
2019-02-13 17:22:49,797 INFO spawned: 'unicorn' with pid 1045
2019-02-13 17:22:51,418 INFO success: unicorn entered RUNNING state, process has stayed up for >

UPDATE Maybe I do not have a memory, but this is not accurate.

ahills60 commented 5 years ago

I too encountered something similar. Investigating the logs by docker exec -it gitlab11 bash into the container and looking at tail log/unicorn.stderr.log revealed that a PID was running or the file /home/git/gitlab/tmp/pids/unicorn.pid is stale. Check for the existence of the PID and, should it not exist, delete the file through rm /home/git/gitlab/tmp/pids/unicorn.pid.

In my instance, I suspect that the container is occasionally crashing/restarting due to not having enough memory and sometimes doesn't clean up after itself following the crash.

geberl commented 5 years ago

I suspect that the container is occasionally crashing/restarting due to not having enough memory and sometimes doesn't clean up after itself following the crash.

For us GitLab crashed after 12minutes during the recompiling assets step that is triggered during startup. This was not the case for 11.8.x, but started at 11.9.x

That whole unicorn issue was a side effect of RAM running out and not the underling cause.

Our solution was to move GitLab from a subpath https://ourdomain.com/apps/gitlab/ to a dedicated subdomain https://git.ourdomain.com/ (by not passing --env GITLAB_RELATIVE_URL_ROOT="/apps/gitlab"). The recompiling assets step is not required at all then. No RAM issues, everything working fine.

Just in case you're also facing this issue @ahills60

ahills60 commented 5 years ago

That makes sense, @geberl, and I'm now investigating a solution that won't involve recompiling assets. Thank you for pointing me in the right direction — it's most appreciated.

Update: Moving to a solution that didn't invoke the recompiling of assets (i.e. removing my GITLAB_RELATIVE_URL_ROOT setup) did the trick and has stopped the GitLab crash. Thanks again @geberl.

wokadeh commented 5 years ago

Is there a fix already? Removing GITLAB_RELATIVE_URL_ROOT is not an option, it is the reason for me to use this project.

wokadeh commented 5 years ago

My bad, my memory was full and this caused the error! All is good again

milovidov983 commented 5 years ago

Most likely the problem was really a lack of memory. But in the end, we installed the original Gitlab image.