ayunwSky
commented
1 year ago I am currently trying to upgrade from gitlab version 14.1.3 to version 15.9.2. I tried to upgrade from 14.1.3 to 15.6.8 without problems, but when I upgraded from 15.6.8 to 15.9.2, gitlab startup error was reported as follows
[root@kube-test-gitlab-srv1 gitlab]# docker logs -f 876d1be73fcb
Loading /etc/docker-gitlab/runtime/env-defaults
Initializing logdir...
Initializing datadir...
Container TimeZone -> Asia/Shanghai
Installing configuration templates...
Configuring gitlab...
Configuring gitlab::database..
Configuring gitlab::redis
Configuring gitlab::actioncable
Configuring gitlab::secrets...
Configuring gitlab::sidekiq...
Configuring gitlab::gitaly...
Configuring gitlab::monitoring...
Configuring gitlab::gitlab-workhorse...
Configuring gitlab::puma...
Configuring gitlab::timezone...
Configuring gitlab::rack_attack...
Configuring gitlab::ci...
Configuring gitlab::artifacts...
Configuring gitlab::packages...
Configuring gitlab::terraform_state...
Configuring gitlab::lfs...
Configuring gitlab::uploads...
Configuring gitlab::mattermost...
Configuring gitlab::project_features...
Configuring gitlab::oauth...
Configuring gitlab::ldap...
Configuring gitlab::cron_jobs...
Configuring gitlab::backups...
Configuring gitlab::backups::schedule...
Configuring gitlab::registry...
Configuring gitlab::pages...
Configuring gitlab::sentry...
Configuring gitlab::content_security_policy...
Configuring gitlab-shell...
Configuring nginx...
Configuring nginx::gitlab...
Migrating database...
2023-03-09 14:18:51,366 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message.
2023-03-09 14:18:51,366 INFO Included extra file "/etc/supervisor/conf.d/cron.conf" during parsing
2023-03-09 14:18:51,367 INFO Included extra file "/etc/supervisor/conf.d/gitaly.conf" during parsing
2023-03-09 14:18:51,367 INFO Included extra file "/etc/supervisor/conf.d/gitlab-workhorse.conf" during parsing
2023-03-09 14:18:51,367 INFO Included extra file "/etc/supervisor/conf.d/groups.conf" during parsing
2023-03-09 14:18:51,368 INFO Included extra file "/etc/supervisor/conf.d/mail_room.conf" during parsing
2023-03-09 14:18:51,368 INFO Included extra file "/etc/supervisor/conf.d/nginx.conf" during parsing
2023-03-09 14:18:51,368 INFO Included extra file "/etc/supervisor/conf.d/puma.conf" during parsing
2023-03-09 14:18:51,369 INFO Included extra file "/etc/supervisor/conf.d/sidekiq.conf" during parsing
2023-03-09 14:18:51,369 INFO Included extra file "/etc/supervisor/conf.d/sshd.conf" during parsing
2023-03-09 14:18:51,394 INFO RPC interface 'supervisor' initialized
2023-03-09 14:18:51,395 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2023-03-09 14:18:51,396 INFO supervisord started with pid 737
2023-03-09 14:18:52,401 INFO spawned: 'gitaly' with pid 757
2023-03-09 14:18:52,407 INFO spawned: 'puma' with pid 758
2023-03-09 14:18:52,413 INFO spawned: 'gitlab-workhorse' with pid 759
2023-03-09 14:18:52,420 INFO spawned: 'sidekiq' with pid 760
2023-03-09 14:18:52,426 INFO spawned: 'sshd' with pid 765
2023-03-09 14:18:52,431 INFO spawned: 'nginx' with pid 766
2023-03-09 14:18:52,438 INFO spawned: 'cron' with pid 770
2023-03-09 14:18:53,999 INFO success: gitaly entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-03-09 14:18:53,999 INFO success: puma entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-03-09 14:18:54,000 INFO success: gitlab-workhorse entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-03-09 14:18:54,000 INFO success: sidekiq entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-03-09 14:18:54,000 INFO success: sshd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-03-09 14:18:54,000 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-03-09 14:18:54,000 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
/home/git/gitlab/lib/gitlab/instrumentation/redis.rb:9: warning: already initialized constant Gitlab::Instrumentation::Redis::ActionCable
/home/git/gitlab/lib/gitlab/instrumentation/redis.rb:9: warning: previous definition of ActionCable was here
rake aborted!
StandardError: An error has occurred, all later migrations canceled:
PG::UndefinedColumn: ERROR: column postgres_foreign_keys.constrained_table_name does not exist
LINE 1: ...ELECT 1 AS one FROM "postgres_foreign_keys" WHERE "postgres_...
^
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:672:in `exec_params'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:672:in `block (2 levels) in exec_no_cache'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.7.2/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.7.2/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:671:in `block in exec_no_cache'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:696:in `block (2 levels) in log'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:695:in `block in log'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.7.2/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:687:in `log'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:670:in `exec_no_cache'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:649:in `execute_and_clear'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/marginalia-1.11.1/lib/marginalia.rb:91:in `execute_and_clear_with_marginalia'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:55:in `exec_query'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:532:in `select'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:69:in `select_all'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:103:in `select_all'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:95:in `select_rows'
/home/git/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:100:in `public_send'
/home/git/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:100:in `block in read_using_load_balancer'
/home/git/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:127:in `block in read_write'
/home/git/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:198:in `retry_with_backoff'
/home/git/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:116:in `read_write'
/home/git/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:99:in `read_using_load_balancer'
/home/git/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:54:in `block (2 levels) in <class:ConnectionProxy>'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/relation/finder_methods.rb:320:in `block in exists?'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/relation.rb:861:in `skip_query_cache_if_necessary'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/relation/finder_methods.rb:320:in `exists?'
/home/git/gitlab/lib/gitlab/database/migration_helpers.rb:390:in `foreign_key_exists?'
/home/git/gitlab/lib/gitlab/database/migration_helpers.rb:1029:in `remove_foreign_key_if_exists'
/home/git/gitlab/db/post_migrate/20221114142044_delete_experiments_foreign_keys.rb:8:in `block in up'
/home/git/gitlab/lib/gitlab/database/with_lock_retries.rb:123:in `run_block'
/home/git/gitlab/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `block in transaction'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.7.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `transaction'
/home/git/gitlab/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout'
/home/git/gitlab/lib/gitlab/database/with_lock_retries.rb:97:in `run'
/home/git/gitlab/lib/gitlab/database/migrations/lock_retries_helpers.rb:52:in `with_lock_retries'
/home/git/gitlab/lib/gitlab/database/migration_helpers/v2.rb:117:in `with_lock_retries'
/home/git/gitlab/db/post_migrate/20221114142044_delete_experiments_foreign_keys.rb:7:in `up'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:870:in `public_send'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:870:in `exec_migration'
/home/git/gitlab/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/home/git/gitlab/lib/gitlab/database/query_analyzer.rb:37:in `within'
/home/git/gitlab/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:851:in `block (2 levels) in migrate'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:850:in `block in migrate'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:849:in `migrate'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1037:in `migrate'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1329:in `block in execute_migration_in_transaction'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1382:in `ddl_transaction'
/home/git/gitlab/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1328:in `execute_migration_in_transaction'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1302:in `each'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1302:in `migrate_without_lock'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1251:in `block in migrate'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1401:in `block in with_advisory_lock'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1416:in `block in with_advisory_lock_connection'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1416:in `with_advisory_lock_connection'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1397:in `with_advisory_lock'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1251:in `migrate'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1086:in `up'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/migration.rb:1061:in `migrate'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/tasks/database_tasks.rb:237:in `migrate'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/railties/databases.rake:92:in `block (3 levels) in <top (required)>'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/railties/databases.rake:90:in `each'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.7.2/lib/active_record/railties/databases.rake:90:in `block (2 levels) in <top (required)>'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/sentry-ruby-core-5.1.1/lib/sentry/rake.rb:26:in `execute'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Caused by:
ActiveRecord::StatementInvalid: PG::UndefinedColumn: ERROR: column postgres_foreign_keys.constrained_table_name does not exist
LINE 1: ...ELECT 1 AS one FROM "postgres_foreign_keys" WHERE "postgres_...Who can help me how to resolve this?
emotroshylov
fidoedidoe
Bouchaib124
Currently I have a GitLab server of 14.1.3 and I would like to upgrade GitLab to 15.8. What is my upgrade path, and what are the precautions of the version upgrade and database migration operation documents? I checked the official gitlab documentation and there was a slight discrepancy between what I saw on the official documentation and what was posted on github