Gave up: nginx entered FATAL state, too many start retries too quickly

[peterwillcn]

Gitlab version 12.2.1-1 https://github.com/sameersbn/docker-gitlab/commit/315b52dee30de84b16c107c121fe3db0f62ca9d6

Configuring nginx::gitlab...
2019-09-07 14:46:28,698 CRIT Supervisor running as root (no user in config file)
2019-09-07 14:46:28,698 INFO Included extra file "/etc/supervisor/conf.d/cron.conf" during parsing
2019-09-07 14:46:28,698 INFO Included extra file "/etc/supervisor/conf.d/gitaly.conf" during parsing
2019-09-07 14:46:28,698 INFO Included extra file "/etc/supervisor/conf.d/gitlab-workhorse.conf" during parsing
2019-09-07 14:46:28,698 INFO Included extra file "/etc/supervisor/conf.d/groups.conf" during parsing
2019-09-07 14:46:28,698 INFO Included extra file "/etc/supervisor/conf.d/mail_room.conf" during parsing
2019-09-07 14:46:28,698 INFO Included extra file "/etc/supervisor/conf.d/nginx.conf" during parsing
2019-09-07 14:46:28,698 INFO Included extra file "/etc/supervisor/conf.d/sidekiq.conf" during parsing
2019-09-07 14:46:28,698 INFO Included extra file "/etc/supervisor/conf.d/sshd.conf" during parsing
2019-09-07 14:46:28,698 INFO Included extra file "/etc/supervisor/conf.d/unicorn.conf" during parsing
2019-09-07 14:46:28,708 INFO RPC interface 'supervisor' initialized
2019-09-07 14:46:28,708 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2019-09-07 14:46:28,708 INFO supervisord started with pid 570
expr: non-integer argument
Migrating database...
2019-09-07 14:46:29,712 INFO spawned: 'gitaly' with pid 591
2019-09-07 14:46:29,716 INFO spawned: 'gitlab-workhorse' with pid 592
2019-09-07 14:46:29,720 INFO spawned: 'unicorn' with pid 593
2019-09-07 14:46:29,724 INFO spawned: 'sidekiq' with pid 594
2019-09-07 14:46:29,727 INFO spawned: 'nginx' with pid 600
2019-09-07 14:46:29,731 INFO spawned: 'sshd' with pid 605
2019-09-07 14:46:29,733 INFO spawned: 'cron' with pid 607
2019-09-07 14:46:29,752 INFO exited: nginx (exit status 1; not expected)
2019-09-07 14:46:31,374 INFO success: gitaly entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-09-07 14:46:31,374 INFO success: gitlab-workhorse entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-09-07 14:46:31,374 INFO success: unicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-09-07 14:46:31,374 INFO success: sidekiq entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-09-07 14:46:31,376 INFO spawned: 'nginx' with pid 631
2019-09-07 14:46:31,376 INFO success: sshd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-09-07 14:46:31,376 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-09-07 14:46:31,396 INFO exited: nginx (exit status 1; not expected)
2019-09-07 14:46:33,407 INFO spawned: 'nginx' with pid 708
2019-09-07 14:46:33,446 INFO exited: nginx (exit status 1; not expected)
2019-09-07 14:46:36,452 INFO spawned: 'nginx' with pid 709
2019-09-07 14:46:36,496 INFO exited: nginx (exit status 1; not expected)
2019-09-07 14:46:37,498 INFO gave up: nginx entered FATAL state, too many start retries too quickly

[arturslogins]

Configuring nginx::gitlab...
2019-09-07 14:46:28,698 CRIT Supervisor running as root (no user in config file)
2019-09-07 14:46:28,698 INFO Included extra file "/etc/supervisor/conf.d/cron.conf" during parsing
2019-09-07 14:46:28,698 INFO Included extra file "/etc/supervisor/conf.d/gitaly.conf" during parsing
2019-09-07 14:46:28,698 INFO Included extra file "/etc/supervisor/conf.d/gitlab-workhorse.conf" during parsing
2019-09-07 14:46:28,698 INFO Included extra file "/etc/supervisor/conf.d/groups.conf" during parsing
2019-09-07 14:46:28,698 INFO Included extra file "/etc/supervisor/conf.d/mail_room.conf" during parsing
2019-09-07 14:46:28,698 INFO Included extra file "/etc/supervisor/conf.d/nginx.conf" during parsing
2019-09-07 14:46:28,698 INFO Included extra file "/etc/supervisor/conf.d/sidekiq.conf" during parsing
2019-09-07 14:46:28,698 INFO Included extra file "/etc/supervisor/conf.d/sshd.conf" during parsing
2019-09-07 14:46:28,698 INFO Included extra file "/etc/supervisor/conf.d/unicorn.conf" during parsing
2019-09-07 14:46:28,708 INFO RPC interface 'supervisor' initialized
2019-09-07 14:46:28,708 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2019-09-07 14:46:28,708 INFO supervisord started with pid 570
expr: non-integer argument
Migrating database...
2019-09-07 14:46:29,712 INFO spawned: 'gitaly' with pid 591
2019-09-07 14:46:29,716 INFO spawned: 'gitlab-workhorse' with pid 592
2019-09-07 14:46:29,720 INFO spawned: 'unicorn' with pid 593
2019-09-07 14:46:29,724 INFO spawned: 'sidekiq' with pid 594
2019-09-07 14:46:29,727 INFO spawned: 'nginx' with pid 600
2019-09-07 14:46:29,731 INFO spawned: 'sshd' with pid 605
2019-09-07 14:46:29,733 INFO spawned: 'cron' with pid 607
2019-09-07 14:46:29,752 INFO exited: nginx (exit status 1; not expected)
2019-09-07 14:46:31,374 INFO success: gitaly entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-09-07 14:46:31,374 INFO success: gitlab-workhorse entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-09-07 14:46:31,374 INFO success: unicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-09-07 14:46:31,374 INFO success: sidekiq entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-09-07 14:46:31,376 INFO spawned: 'nginx' with pid 631
2019-09-07 14:46:31,376 INFO success: sshd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-09-07 14:46:31,376 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-09-07 14:46:31,396 INFO exited: nginx (exit status 1; not expected)
2019-09-07 14:46:33,407 INFO spawned: 'nginx' with pid 708
2019-09-07 14:46:33,446 INFO exited: nginx (exit status 1; not expected)
2019-09-07 14:46:36,452 INFO spawned: 'nginx' with pid 709
2019-09-07 14:46:36,496 INFO exited: nginx (exit status 1; not expected)
2019-09-07 14:46:37,498 INFO gave up: nginx entered FATAL state, too many start retries too quickly

Gitlab version?

[peterwillcn]

Gitlab version? @arturslogins 12.2.1-1 https://github.com/sameersbn/docker-gitlab/commit/315b52dee30de84b16c107c121fe3db0f62ca9d6

[wflanagan]

12.2.5 version published 5 days ago has this same problem on upgrade..

[SzymonDukla]

Yup, having problems to boot and attach to nginx proxy on host on latest version (12.2.5)

[paddy-hack]

Encountered something very similar this morning but in my case nginx just kept restarting. A docker-compose down; docker-compose up -d didn't fix that, at least not the first time around. I tried removing a stale /var/run/nginx.pid but that didn't fix it. One or two docker-compose down/up cycles later nginx started and the gitlab container was fine again.

This looks like a timing related issue.

BTW, the nginx reverse proxy that I have in front of my gitlab container reported a pile of Connection refused errors, then started reporting Host unreachable errors. A nginx -s reload fixed that for me. I'm not sure about what was going on but things work for me now :cold_sweat:

[darkautism]

12.5.2 still has this problem

Edit: Because my dns cannot resolve registry. It should been fixed after DNS server has been setup.

[TheWiresharkGuy]

Any fix yet ?

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity for the last 60 days. It will be closed if no further activity occurs during the next 7 days. Thank you for your contributions.