sameersbn / docker-gitlab

Dockerized GitLab
http://www.damagehead.com/docker-gitlab/
MIT License
7.88k stars 2.14k forks source link

Unable to upgrade/launch gitlab image past 12.3 #2036

Closed Makosso closed 4 years ago

Makosso commented 4 years ago

Hi! I've been using this image for quite some time, however it refuses to upgrade, or even launch with no data from image version 12.3 and up. The error I'm getting is:

gitlab_1 | 2019-10-24 10:41:09,387 CRIT Supervisor running as root (no user in config file) gitlab_1 | 2019-10-24 10:41:09,387 INFO Included extra file "/etc/supervisor/conf.d/cron.conf" during parsing gitlab_1 | 2019-10-24 10:41:09,387 INFO Included extra file "/etc/supervisor/conf.d/gitaly.conf" during parsing gitlab_1 | 2019-10-24 10:41:09,387 INFO Included extra file "/etc/supervisor/conf.d/gitlab-workhorse.conf" during parsing gitlab_1 | 2019-10-24 10:41:09,387 INFO Included extra file "/etc/supervisor/conf.d/groups.conf" during parsing gitlab_1 | 2019-10-24 10:41:09,387 INFO Included extra file "/etc/supervisor/conf.d/mail_room.conf" during parsing gitlab_1 | 2019-10-24 10:41:09,387 INFO Included extra file "/etc/supervisor/conf.d/nginx.conf" during parsing gitlab_1 | 2019-10-24 10:41:09,387 INFO Included extra file "/etc/supervisor/conf.d/sidekiq.conf" during parsing gitlab_1 | 2019-10-24 10:41:09,387 INFO Included extra file "/etc/supervisor/conf.d/sshd.conf" during parsing gitlab_1 | 2019-10-24 10:41:09,387 INFO Included extra file "/etc/supervisor/conf.d/unicorn.conf" during parsing gitlab_1 | 2019-10-24 10:41:09,398 INFO RPC interface 'supervisor' initialized gitlab_1 | 2019-10-24 10:41:09,398 CRIT Server 'unix_http_server' running without any HTTP authentication checking gitlab_1 | 2019-10-24 10:41:09,398 INFO supervisord started with pid 635 gitlab_1 | 2019-10-24 10:41:10,401 INFO spawned: 'gitaly' with pid 649 gitlab_1 | 2019-10-24 10:41:10,403 INFO spawned: 'gitlab-workhorse' with pid 650 gitlab_1 | 2019-10-24 10:41:10,404 INFO spawned: 'unicorn' with pid 651 gitlab_1 | 2019-10-24 10:41:10,406 INFO spawned: 'sidekiq' with pid 652 gitlab_1 | 2019-10-24 10:41:10,408 INFO spawned: 'mail_room' with pid 653 gitlab_1 | 2019-10-24 10:41:10,410 INFO spawned: 'nginx' with pid 654 gitlab_1 | 2019-10-24 10:41:10,412 INFO spawned: 'sshd' with pid 655 gitlab_1 | 2019-10-24 10:41:10,414 INFO spawned: 'cron' with pid 666 gitlab_1 | 2019-10-24 10:41:11,500 INFO success: gitaly entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) gitlab_1 | 2019-10-24 10:41:11,500 INFO success: gitlab-workhorse entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) gitlab_1 | 2019-10-24 10:41:11,500 INFO success: unicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) gitlab_1 | 2019-10-24 10:41:11,500 INFO success: sidekiq entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) gitlab_1 | 2019-10-24 10:41:11,500 INFO success: mail_room entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) gitlab_1 | 2019-10-24 10:41:11,500 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) gitlab_1 | 2019-10-24 10:41:11,500 INFO success: sshd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) gitlab_1 | 2019-10-24 10:41:11,500 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) gitlab_1 | Missing Rails.application.secrets.openid_connect_signing_key for production environment. The secret will be generated and stored in config/secrets.yml. postgresql_1 | ERROR: relation "feature_gates" does not exist at character 566 postgresql_1 | STATEMENT: SELECT a.attname, format_type(a.atttypid, a.atttypmod), postgresql_1 | pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod, postgresql_1 | c.collname, col_description(a.attrelid, a.attnum) AS comment postgresql_1 | FROM pg_attribute a postgresql_1 | LEFT JOIN pg_attrdef d ON a.attrelid = d.adrelid AND a.attnum = d.adnum postgresql_1 | LEFT JOIN pg_type t ON a.atttypid = t.oid postgresql_1 | LEFT JOIN pg_collation c ON a.attcollation = c.oid AND a.attcollation <> t.typcollation postgresql_1 | WHERE a.attrelid = '"feature_gates"'::regclass postgresql_1 | AND a.attnum > 0 AND NOT a.attisdropped postgresql_1 | ORDER BY a.attnum postgresql_1 |
gitlab_1 | rake aborted! gitlab_1 | storage "default" is missing a gitaly_address gitlab_1 | /home/git/gitlab/lib/gitlab/gitaly_client.rb:113:in address' gitlab_1 | /home/git/gitlab/config/initializers/8_gitaly.rb:5:inblock in <top (required)>' gitlab_1 | /home/git/gitlab/config/initializers/8_gitaly.rb:3:in each' gitlab_1 | /home/git/gitlab/config/initializers/8_gitaly.rb:3:in<top (required)>' gitlab_1 | /home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:285:in load' gitlab_1 | /home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:285:inblock in load' gitlab_1 | /home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:257:in load_dependency' gitlab_1 | /home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:285:inload' gitlab_1 | /home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/engine.rb:657:in block in load_config_initializer' gitlab_1 | /home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/notifications.rb:170:ininstrument' gitlab_1 | /home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/engine.rb:656:in load_config_initializer' gitlab_1 | /home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/engine.rb:614:inblock (2 levels) in ' gitlab_1 | /home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/engine.rb:613:in each' gitlab_1 | /home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/engine.rb:613:inblock in ' gitlab_1 | /home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/initializable.rb:32:in instance_exec' gitlab_1 | /home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/initializable.rb:32:inrun' gitlab_1 | /home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/initializable.rb:61:in block in run_initializers' gitlab_1 | /home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/initializable.rb:50:ineach' gitlab_1 | /home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/initializable.rb:50:in tsort_each_child' gitlab_1 | /home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/initializable.rb:60:inrun_initializers' gitlab_1 | /home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/application.rb:361:in initialize!' gitlab_1 | /home/git/gitlab/config/environment.rb:6:in<top (required)>' gitlab_1 | /home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:291:in require' gitlab_1 | /home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:291:inblock in require' gitlab_1 | /home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:257:in load_dependency' gitlab_1 | /home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:291:inrequire' gitlab_1 | /home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/application.rb:337:in require_environment!' gitlab_1 | /home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/application.rb:520:inblock in run_tasks_blocks' gitlab_1 | /home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/exe/rake:27:in `<top (required)>' gitlab_1 | Tasks: TOP => gitlab:setup => gitlab_environment => environment gitlab_1 | (See full trace by running task with --trace) gitlab_gitlab exited with code 1

The same thing happens (rake aborted/storage default is missing a gitaly_address) both when upgrading the image on previously working instance (tried upgrading from 11.11.0 and 12.1.6) and on a fresh deploy, with no data. My current instance based on 12.1.6 image works with no problem whatsoever.

os: ubuntu 18.04 4.15.0-34-generic docker version: 18.06.1-ce, build e68fc7a docker-compose version: 1.17.1, build unknown

Any clue as to what's the cause of this behavior? Is there a particular thing I need to fix on my end in order to upgrade to 12.3.x and further?

danyill commented 4 years ago

I have a similar problem. I think in the docker-compose file it is caused by the moved away from bind mounts and towards volumes (see https://github.com/sameersbn/docker-gitlab/issues/2027). There may also be other things as I was unable to upgrade from 12.1.6 to 12.3.5 even after reverting to bind mounts (which I have yet to diagnose) but you could try that.

Makosso commented 4 years ago

@danyill thanks for the tip, however using volumes in place of binds doesn't change anything. On a fresh instance, using docker volumes I'm still getting the same error, resulting in gitlab container crashing with exit code 1:

gitlab_1 | rake aborted! gitlab_1 | storage "default" is missing a gitaly_address

danyill commented 4 years ago

I found I could migrate from 12.1.6 to 12.5.2 today without any difficulty so I suspect there was an issue in perhaps the 12.3 series.

I've retained bind mounts (so changed the docker-compose.yml only from the one available in this repo around about the tag 12.1.6) and then only changed the version of the gitlab container from 12.1.6 to 12.5.2 to achieve this).

Compared to the current docker-compose.yml, for the bind mounts, the changes are something like this:

10c10
<     - /srv/docker/gitlab/redis:/var/lib/redis:Z
---
>     - redis-data:/var/lib/redis:Z
16c16
<     - /srv/docker/gitlab/postgresql:/var/lib/postgresql:Z
---
>     - postgresql-data:/var/lib/postgresql:Z
34c33
<     - /srv/docker/gitlab/gitlab:/home/git/data:Z
---
>     - gitlab-data:/home/git/data:Z
> volumes:
>   redis-data:
>   postgresql-data:
>   gitlab-data:

and from:

  gitlab:
    restart: always
    image: sameersbn/gitlab:12.1.6

to:

  gitlab:
    restart: always
    image: sameersbn/gitlab:12.5.2
Makosso commented 4 years ago

Thank You for the mention @danyill, will definitely try the new image once I get back from holiday

f3l1x commented 4 years ago

Hi. I've just run into the same issue.

gitlab_1      | 2020-01-31 20:08:04,817 INFO success: unicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
gitlab_1      | 2020-01-31 20:08:04,817 INFO success: sidekiq entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
gitlab_1      | 2020-01-31 20:08:04,817 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
gitlab_1      | 2020-01-31 20:08:04,817 INFO success: sshd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
gitlab_1      | 2020-01-31 20:08:04,818 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
postgresql_1  | 2020-01-31 19:09:00.693 UTC [220] ERROR:  relation "feature_gates" does not exist at character 566
postgresql_1  | 2020-01-31 19:09:00.693 UTC [220] STATEMENT:                SELECT a.attname, format_type(a.atttypid, a.atttypmod),
postgresql_1  |                          pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod,
postgresql_1  |                          c.collname, col_description(a.attrelid, a.attnum) AS comment
postgresql_1  |                     FROM pg_attribute a
postgresql_1  |                     LEFT JOIN pg_attrdef d ON a.attrelid = d.adrelid AND a.attnum = d.adnum
postgresql_1  |                     LEFT JOIN pg_type t ON a.atttypid = t.oid
postgresql_1  |                     LEFT JOIN pg_collation c ON a.attcollation = c.oid AND a.attcollation <> t.typcollation
postgresql_1  |                    WHERE a.attrelid = '"feature_gates"'::regclass
postgresql_1  |                      AND a.attnum > 0 AND NOT a.attisdropped
postgresql_1  |                    ORDER BY a.attnum
postgresql_1  |
postgresql_1  | 2020-01-31 19:09:03.586 UTC [221] ERROR:  relation "feature_gates" does not exist at character 566
postgresql_1  | 2020-01-31 19:09:03.586 UTC [221] STATEMENT:                SELECT a.attname, format_type(a.atttypid, a.atttypmod),
postgresql_1  |                          pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod,
postgresql_1  |                          c.collname, col_description(a.attrelid, a.attnum) AS comment
postgresql_1  |                     FROM pg_attribute a
postgresql_1  |                     LEFT JOIN pg_attrdef d ON a.attrelid = d.adrelid AND a.attnum = d.adnum
postgresql_1  |                     LEFT JOIN pg_type t ON a.atttypid = t.oid
postgresql_1  |                     LEFT JOIN pg_collation c ON a.attcollation = c.oid AND a.attcollation <> t.typcollation
postgresql_1  |                    WHERE a.attrelid = '"feature_gates"'::regclass
postgresql_1  |                      AND a.attnum > 0 AND NOT a.attisdropped
postgresql_1  |                    ORDER BY a.attnum
postgresql_1  |
postgresql_1  | 2020-01-31 19:09:04.602 UTC [222] ERROR:  relation "feature_gates" does not exist at character 566
postgresql_1  | 2020-01-31 19:09:04.602 UTC [222] STATEMENT:                SELECT a.attname, format_type(a.atttypid, a.atttypmod),
postgresql_1  |                          pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod,
postgresql_1  |                          c.collname, col_description(a.attrelid, a.attnum) AS comment
postgresql_1  |                     FROM pg_attribute a
postgresql_1  |                     LEFT JOIN pg_attrdef d ON a.attrelid = d.adrelid AND a.attnum = d.adnum
postgresql_1  |                     LEFT JOIN pg_type t ON a.atttypid = t.oid
postgresql_1  |                     LEFT JOIN pg_collation c ON a.attcollation = c.oid AND a.attcollation <> t.typcollation
postgresql_1  |                    WHERE a.attrelid = '"feature_gates"'::regclass
postgresql_1  |                      AND a.attnum > 0 AND NOT a.attisdropped
postgresql_1  |                    ORDER BY a.attnum
postgresql_1  |
gitlab_1      | Database 'gitlabhq_production' already exists
postgresql_1  | 2020-01-31 19:09:13.633 UTC [226] ERROR:  database "gitlabhq_production" already exists
postgresql_1  | 2020-01-31 19:09:13.633 UTC [226] STATEMENT:  CREATE DATABASE "gitlabhq_production" ENCODING = 'unicode'
gitlab_1      | Migrating database...
gitlab_1      | Clearing cache...
gitlab_1      | 2020-01-31 20:11:42,427 WARN received SIGTERM indicating exit request
gitlab_1      | 2020-01-31 20:11:42,433 INFO waiting for gitaly, gitlab-workhorse, unicorn, sidekiq, nginx, sshd, cron to die
gitlab_1      | 2020-01-31 20:11:42,434 INFO stopped: sshd (exit status 0)
gitlab_1      | 2020-01-31 20:11:42,434 INFO stopped: cron (terminated by SIGTERM)

I've took docker-compose.yml and changed volumes.

  gitlab:
    restart: always
    image: sameersbn/gitlab:12.7.0
    depends_on:
    - redis
    - postgresql
    ports:
    - "22:22"
    volumes:
    - ./data/gitlab:/home/git/data:Z
f3l1x commented 4 years ago

Well, it works eventually even when error is showed. Weird. :-)

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had any activity for the last 60 days. It will be closed if no further activity occurs during the next 7 days. Thank you for your contributions.