search

sameersbn / docker-gitlab

Dockerized GitLab
http://www.damagehead.com/docker-gitlab/
MIT License
7.87k stars 2.14k forks source link

Upgrade from 14.1.3 (and prior) to 14.2.1 fails due to database migration issue #2409

Open crud42 opened 3 years ago

crud42 commented 3 years ago

Currently i experience an error when trying to upgrade to 14.2.1 form any 13.x and 14.x Version.

The message prior to the error:

_Caused by: gitlab_1 | Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active': {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"ci_job_artifacts", :column_name=>"id", :job_arguments=>[["id", "job_id"], ["id_convert_to_bigint", "job_id_convert_to_bigint"]]} gitlab_1 | gitlab_1 | Finalize it manualy by running gitlab_1 | gitlab_1 | sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_job_artifacts,id,'[["id"\, "job_id"]\, ["id_convert_to_bigint"\, "job_id_convert_tobigint"]]']

I tracked the error down and found a hint at

https://docs.gitlab.com/ee/update/#1410 mentioning the migration issue.

Just wanted to point that out, maybe there is a patch missing?

Best, Chris

Gaibhne commented 3 years ago

Did you manually pull the repo commits ? From what I can see, the latest release of this repo is still 14.1.1-1, which might be because of the issue you are facing. If so it might be helpful to add to this issue which commit you were at and to which commit you attempted to migrate.

crud42 commented 3 years ago

Hi! No i did not pull manually, i upgraded through docker image sameersbn/gitlab:14.2.1. I am currently on docker image sameersbn/gitlab:14.1.3.

Maybe i got something wrong, but usually pulling from the corresponding tagged image worked. But maybe its not equal to the gitlab code release?

kkimurak commented 3 years ago

@crud42 Yes, release of sameersbn/gitlab is NOT equal to the gitlab's code release. Pull Request is submitted by someone as soon as possible after code release, but it takes few weeks to be merged after gitlab's release in worst case.
Note that some tags will not work because the release is not created (e.g. sameersbn/gitlab:14.0.6 is final releases of v14.0.Z so 14.0.7 and later is not available).

@Gaibhne sameersbn/gitlab:14.2.1 is available on docker hub and quay.io, but release seems to be failing.

@sameersbn Could you please take a look for Circle-CI configuration? It says "github returned 401 Unauthorized" during github-release command (so I think problem is around Access Token or something)

Kaleesastha commented 3 years ago

We also facing an issue in migration with 14.2.2 from 14.1.3/14.2.1. The exception logs are given below

2021-09-02T08:08:16.346983000Z Migrating database... 2021-09-02T08:08:16.538752000Z 2021-09-02 13:38:16,537 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message. 2021-09-02T08:08:16.539390000Z 2021-09-02 13:38:16,538 INFO Included extra file "/etc/supervisor/conf.d/cron.conf" during parsing 2021-09-02T08:08:16.539859000Z 2021-09-02 13:38:16,538 INFO Included extra file "/etc/supervisor/conf.d/gitaly.conf" during parsing 2021-09-02T08:08:16.540342000Z 2021-09-02 13:38:16,538 INFO Included extra file "/etc/supervisor/conf.d/gitlab-pages.conf" during parsing 2021-09-02T08:08:16.540791000Z 2021-09-02 13:38:16,538 INFO Included extra file "/etc/supervisor/conf.d/gitlab-workhorse.conf" during parsing 2021-09-02T08:08:16.541246000Z 2021-09-02 13:38:16,538 INFO Included extra file "/etc/supervisor/conf.d/groups.conf" during parsing 2021-09-02T08:08:16.541723000Z 2021-09-02 13:38:16,538 INFO Included extra file "/etc/supervisor/conf.d/mail_room.conf" during parsing 2021-09-02T08:08:16.542274000Z 2021-09-02 13:38:16,538 INFO Included extra file "/etc/supervisor/conf.d/nginx.conf" during parsing 2021-09-02T08:08:16.542782000Z 2021-09-02 13:38:16,538 INFO Included extra file "/etc/supervisor/conf.d/puma.conf" during parsing 2021-09-02T08:08:16.543346000Z 2021-09-02 13:38:16,538 INFO Included extra file "/etc/supervisor/conf.d/sidekiq.conf" during parsing 2021-09-02T08:08:16.543858000Z 2021-09-02 13:38:16,538 INFO Included extra file "/etc/supervisor/conf.d/sshd.conf" during parsing 2021-09-02T08:08:16.545021000Z 2021-09-02 13:38:16,544 INFO RPC interface 'supervisor' initialized 2021-09-02T08:08:16.545594000Z 2021-09-02 13:38:16,544 CRIT Server 'unix_http_server' running without any HTTP authentication checking 2021-09-02T08:08:16.546210000Z 2021-09-02 13:38:16,545 INFO supervisord started with pid 789 2021-09-02T08:08:17.548779000Z 2021-09-02 13:38:17,547 INFO spawned: 'gitaly' with pid 809 2021-09-02T08:08:17.550547000Z 2021-09-02 13:38:17,549 INFO spawned: 'puma' with pid 810 2021-09-02T08:08:17.552892000Z 2021-09-02 13:38:17,552 INFO spawned: 'gitlab-workhorse' with pid 811 2021-09-02T08:08:17.555204000Z 2021-09-02 13:38:17,554 INFO spawned: 'sidekiq' with pid 812 2021-09-02T08:08:17.557257000Z 2021-09-02 13:38:17,556 INFO spawned: 'gitlab-pages' with pid 813 2021-09-02T08:08:17.559680000Z 2021-09-02 13:38:17,558 INFO spawned: 'sshd' with pid 814 2021-09-02T08:08:17.562449000Z 2021-09-02 13:38:17,561 INFO spawned: 'nginx' with pid 815 2021-09-02T08:08:17.564854000Z 2021-09-02 13:38:17,564 INFO spawned: 'cron' with pid 816 2021-09-02T08:08:18.566797000Z 2021-09-02 13:38:18,566 INFO success: gitaly entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2021-09-02T08:08:18.567281000Z 2021-09-02 13:38:18,566 INFO success: puma entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2021-09-02T08:08:18.567659000Z 2021-09-02 13:38:18,566 INFO success: gitlab-workhorse entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2021-09-02T08:08:18.568073000Z 2021-09-02 13:38:18,566 INFO success: sidekiq entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2021-09-02T08:08:18.568481000Z 2021-09-02 13:38:18,566 INFO success: gitlab-pages entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2021-09-02T08:08:18.568892000Z 2021-09-02 13:38:18,566 INFO success: sshd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2021-09-02T08:08:18.569297000Z 2021-09-02 13:38:18,566 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2021-09-02T08:08:18.569765000Z 2021-09-02 13:38:18,566 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2021-09-02T08:08:21.312201000Z 2021-09-02 13:38:21,311 INFO exited: gitlab-pages (exit status 1; not expected) 2021-09-02T08:08:21.371915000Z 2021-09-02 13:38:21,371 INFO spawned: 'gitlab-pages' with pid 855 2021-09-02T08:08:21.400598000Z 2021-09-02 13:38:21,399 INFO exited: gitlab-pages (exit status 1; not expected) 2021-09-02T08:08:22.425424000Z 2021-09-02 13:38:22,424 INFO spawned: 'gitlab-pages' with pid 871 2021-09-02T08:08:22.450601000Z 2021-09-02 13:38:22,449 INFO exited: gitlab-pages (exit status 1; not expected) 2021-09-02T08:08:24.951829000Z 2021-09-02 13:38:24,950 INFO spawned: 'gitlab-pages' with pid 880 2021-09-02T08:08:24.982039000Z 2021-09-02 13:38:24,981 INFO exited: gitlab-pages (exit status 1; not expected) 2021-09-02T08:08:27.986971000Z 2021-09-02 13:38:27,986 INFO spawned: 'gitlab-pages' with pid 965 2021-09-02T08:08:28.016529000Z 2021-09-02 13:38:28,015 INFO exited: gitlab-pages (exit status 1; not expected) 2021-09-02T08:08:29.018180000Z 2021-09-02 13:38:29,017 INFO gave up: gitlab-pages entered FATAL state, too many start retries too quickly 2021-09-02T08:09:04.263578000Z rake aborted! 2021-09-02T08:09:04.264173000Z StandardError: An error has occurred, all later migrations canceled: 2021-09-02T08:09:04.265088000Z Table 'push_event_payloads' has no foreign key for {:name=>"fk_36c74129da"} 2021-09-02T08:09:04.265474000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/connection_adapters/abstract/schema_statements.rb:1504:in foreign_key_for!' 2021-09-02T08:09:04.265918000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/connection_adapters/abstract/schema_statements.rb:1097:inremove_foreign_key' 2021-09-02T08:09:04.266340000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:929:in block in method_missing' 2021-09-02T08:09:04.266774000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:897:inblock in say_with_time' 2021-09-02T08:09:04.267246000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:897:in say_with_time' 2021-09-02T08:09:04.268099000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:918:inmethod_missing' 2021-09-02T08:09:04.268691000Z /home/git/gitlab/db/post_migrate/20210802043253_finalize_push_event_payloads_bigint_conversion_3.rb:77:in block in swap_columns' 2021-09-02T08:09:04.269274000Z /home/git/gitlab/lib/gitlab/database/with_lock_retries.rb:121:inrun_block' 2021-09-02T08:09:04.269847000Z /home/git/gitlab/lib/gitlab/database/with_lock_retries.rb:130:in block in run_block_with_lock_timeout' 2021-09-02T08:09:04.270416000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/connection_adapters/abstract/database_statements.rb:320:inblock in transaction' 2021-09-02T08:09:04.270998000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/connection_adapters/abstract/transaction.rb:310:in block in within_new_transaction' 2021-09-02T08:09:04.271576000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:inblock (2 levels) in synchronize' 2021-09-02T08:09:04.272165000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in handle_interrupt' 2021-09-02T08:09:04.272727000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:inblock in synchronize' 2021-09-02T08:09:04.273302000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in handle_interrupt' 2021-09-02T08:09:04.273876000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:insynchronize' 2021-09-02T08:09:04.285186000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/connection_adapters/abstract/transaction.rb:308:in within_new_transaction' 2021-09-02T08:09:04.285870000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/connection_adapters/abstract/database_statements.rb:320:intransaction' 2021-09-02T08:09:04.286459000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/transactions.rb:209:in transaction' 2021-09-02T08:09:04.287048000Z /home/git/gitlab/lib/gitlab/database.rb:214:inblock in transaction' 2021-09-02T08:09:04.287615000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/notifications.rb:205:in instrument' 2021-09-02T08:09:04.288181000Z /home/git/gitlab/lib/gitlab/database.rb:213:intransaction' 2021-09-02T08:09:04.288743000Z /home/git/gitlab/lib/gitlab/database/with_lock_retries.rb:125:in run_block_with_lock_timeout' 2021-09-02T08:09:04.289302000Z /home/git/gitlab/lib/gitlab/database/with_lock_retries.rb:95:inrun' 2021-09-02T08:09:04.289864000Z /home/git/gitlab/lib/gitlab/database/migration_helpers.rb:418:in with_lock_retries' 2021-09-02T08:09:04.290420000Z /home/git/gitlab/db/post_migrate/20210802043253_finalize_push_event_payloads_bigint_conversion_3.rb:45:inswap_columns' 2021-09-02T08:09:04.290986000Z /home/git/gitlab/db/post_migrate/20210802043253_finalize_push_event_payloads_bigint_conversion_3.rb:21:in up' 2021-09-02T08:09:04.291567000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:870:inpublic_send' 2021-09-02T08:09:04.292157000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:870:in exec_migration' 2021-09-02T08:09:04.292744000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:851:inblock (2 levels) in migrate' 2021-09-02T08:09:04.302070000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:850:in block in migrate' 2021-09-02T08:09:04.302641000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:inwith_connection' 2021-09-02T08:09:04.304410000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:849:in migrate' 2021-09-02T08:09:04.305006000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1037:inmigrate' 2021-09-02T08:09:04.305600000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1329:in block in execute_migration_in_transaction' 2021-09-02T08:09:04.306177000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1382:inddl_transaction' 2021-09-02T08:09:04.306753000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1328:in execute_migration_in_transaction' 2021-09-02T08:09:04.307368000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1302:ineach' 2021-09-02T08:09:04.307915000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1302:in migrate_without_lock' 2021-09-02T08:09:04.308457000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1251:inblock in migrate' 2021-09-02T08:09:04.309050000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1401:in block in with_advisory_lock' 2021-09-02T08:09:04.309647000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1416:inblock in with_advisory_lock_connection' 2021-09-02T08:09:04.310255000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in with_connection' 2021-09-02T08:09:04.310869000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1416:inwith_advisory_lock_connection' 2021-09-02T08:09:04.311468000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1397:in with_advisory_lock' 2021-09-02T08:09:04.312073000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1251:inmigrate' 2021-09-02T08:09:04.312648000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1086:in up' 2021-09-02T08:09:04.313243000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1061:inmigrate' 2021-09-02T08:09:04.313799000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/tasks/database_tasks.rb:237:in migrate' 2021-09-02T08:09:04.314411000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/railties/databases.rake:92:inblock (3 levels) in <top (required)>' 2021-09-02T08:09:04.314996000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/railties/databases.rake:90:in each' 2021-09-02T08:09:04.315532000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/railties/databases.rake:90:inblock (2 levels) in <top (required)>' 2021-09-02T08:09:04.327235000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/exe/rake:27:in <top (required)>' 2021-09-02T08:09:04.328614000Z Caused by: 2021-09-02T08:09:04.329237000Z ArgumentError: Table 'push_event_payloads' has no foreign key for {:name=>"fk_36c74129da"} 2021-09-02T08:09:04.329828000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/connection_adapters/abstract/schema_statements.rb:1504:inforeign_key_for!' 2021-09-02T08:09:04.330418000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/connection_adapters/abstract/schema_statements.rb:1097:in remove_foreign_key' 2021-09-02T08:09:04.331003000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:929:inblock in method_missing' 2021-09-02T08:09:04.331597000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:897:in block in say_with_time' 2021-09-02T08:09:04.332173000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:897:insay_with_time' 2021-09-02T08:09:04.332759000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:918:in method_missing' 2021-09-02T08:09:04.339157000Z /home/git/gitlab/db/post_migrate/20210802043253_finalize_push_event_payloads_bigint_conversion_3.rb:77:inblock in swap_columns' 2021-09-02T08:09:04.340528000Z /home/git/gitlab/lib/gitlab/database/with_lock_retries.rb:121:in run_block' 2021-09-02T08:09:04.341843000Z /home/git/gitlab/lib/gitlab/database/with_lock_retries.rb:130:inblock in run_block_with_lock_timeout' 2021-09-02T08:09:04.343172000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in block in transaction' 2021-09-02T08:09:04.344429000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/connection_adapters/abstract/transaction.rb:310:inblock in within_new_transaction' 2021-09-02T08:09:04.345578000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in block (2 levels) in synchronize' 2021-09-02T08:09:04.346774000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:inhandle_interrupt' 2021-09-02T08:09:04.348123000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in block in synchronize' 2021-09-02T08:09:04.349335000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:inhandle_interrupt' 2021-09-02T08:09:04.350533000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in synchronize' 2021-09-02T08:09:04.351681000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/connection_adapters/abstract/transaction.rb:308:inwithin_new_transaction' 2021-09-02T08:09:04.352908000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in transaction' 2021-09-02T08:09:04.354118000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/transactions.rb:209:intransaction' 2021-09-02T08:09:04.355335000Z /home/git/gitlab/lib/gitlab/database.rb:214:in block in transaction' 2021-09-02T08:09:04.356496000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/notifications.rb:205:ininstrument' 2021-09-02T08:09:04.357722000Z /home/git/gitlab/lib/gitlab/database.rb:213:in transaction' 2021-09-02T08:09:04.358800000Z /home/git/gitlab/lib/gitlab/database/with_lock_retries.rb:125:inrun_block_with_lock_timeout' 2021-09-02T08:09:04.359358000Z /home/git/gitlab/lib/gitlab/database/with_lock_retries.rb:95:in run' 2021-09-02T08:09:04.359932000Z /home/git/gitlab/lib/gitlab/database/migration_helpers.rb:418:inwith_lock_retries' 2021-09-02T08:09:04.360512000Z /home/git/gitlab/db/post_migrate/20210802043253_finalize_push_event_payloads_bigint_conversion_3.rb:45:in swap_columns' 2021-09-02T08:09:04.361124000Z /home/git/gitlab/db/post_migrate/20210802043253_finalize_push_event_payloads_bigint_conversion_3.rb:21:inup' 2021-09-02T08:09:04.361768000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:870:in public_send' 2021-09-02T08:09:04.362333000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:870:inexec_migration' 2021-09-02T08:09:04.362979000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:851:in block (2 levels) in migrate' 2021-09-02T08:09:04.363519000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:850:inblock in migrate' 2021-09-02T08:09:04.364100000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in with_connection' 2021-09-02T08:09:04.365477000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:849:inmigrate' 2021-09-02T08:09:04.366018000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1037:in migrate' 2021-09-02T08:09:04.366522000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1329:inblock in execute_migration_in_transaction' 2021-09-02T08:09:04.367067000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1382:in ddl_transaction' 2021-09-02T08:09:04.367646000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1328:inexecute_migration_in_transaction' 2021-09-02T08:09:04.368421000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1302:in each' 2021-09-02T08:09:04.369077000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1302:inmigrate_without_lock' 2021-09-02T08:09:04.369720000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1251:in block in migrate' 2021-09-02T08:09:04.370349000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1401:inblock in with_advisory_lock' 2021-09-02T08:09:04.371005000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1416:in block in with_advisory_lock_connection' 2021-09-02T08:09:04.372111000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:inwith_connection' 2021-09-02T08:09:04.372719000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1416:in with_advisory_lock_connection' 2021-09-02T08:09:04.373778000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1397:inwith_advisory_lock' 2021-09-02T08:09:04.374317000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1251:in migrate' 2021-09-02T08:09:04.375424000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1086:inup' 2021-09-02T08:09:04.375972000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1061:in migrate' 2021-09-02T08:09:04.376609000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/tasks/database_tasks.rb:237:inmigrate' 2021-09-02T08:09:04.377161000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/railties/databases.rake:92:in block (3 levels) in <top (required)>' 2021-09-02T08:09:04.378089000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/railties/databases.rake:90:ineach' 2021-09-02T08:09:04.378721000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/railties/databases.rake:90:in block (2 levels) in <top (required)>' 2021-09-02T08:09:04.379959000Z /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/exe/rake:27:in<top (required)>' 2021-09-02T08:09:04.380491000Z Tasks: TOP => db:migrate 2021-09-02T08:09:04.381487000Z (See full trace by running task with --trace)

Kindly let us share the fix for this issue. We are awaiting for your fix.

EugRomanchenko commented 3 years ago

@crud42 what version of the database are you using? https://docs.gitlab.com/ee/install/requirements.html#database

The support of PostgreSQL 11 has been dropped. Make sure to update your database to version 12 before updating to GitLab 14.0.

Kaleesastha commented 3 years ago

We are awaiting for your reply. We are using PostgreSQL 13. Without v14.2.0, we couldn't proceed the further version migrations. Also, we have an issue in sending the mail for reset password, job status, merge request commit's.

@sameersbn Can you please check the issues and provide the fix?

sameersbn commented 3 years ago

@Kaleesastha thanks for bringing this to my attention. I will take a look and try to figure out the fix

f3l1x commented 3 years ago

I hit exactly the same issue. Upgrading from gitlab (14.0.2 => 14.2.4), postres (sameersbn/postgresql:12-20200524) and redis (6.2).

crud42 commented 3 years ago

@crud42 what version of the database are you using? https://docs.gitlab.com/ee/install/requirements.html#database

The support of PostgreSQL 11 has been dropped. Make sure to update your database to version 12 before updating to GitLab 14.0.

@EugRomanchenko Already on PostgreSQL 12 for a while now.

Still on 14.1.3 - i am currently uncertain if i should update

richardland commented 3 years ago

Only thing that actually worked for me was rebooting the server (even restarting gitlab incl postgres did not do the trick and lsof did show version 12 of postgres running).

After the reboot I re-ran gitlab-ctl reconfigure and everything was OK. Upgrade from 14.2 to 14.3 then also went smooth.

P.S. Running Debian 10.10

cloudnotify commented 2 years ago

@sameersbn Hope, you're working on it. Shall I know when it will get completed / be resolved?

u03c1 commented 2 years ago

We faced the same issues as @crud42 about Migrations. We were previously on a old 13.2 with Postgres 11.

We finally succeed with this workflow. 1) Upgrade to Postgres 12, wait postgres to auto migrate data on 12 directory, then reup gitlab. 2) Check and fix missing hashed container ID with docker exec --user git -it own-gitlab bundle exec rake gitlab:storage:migrate_to_hashed RAILS_ENV=production (It's unrelated with current issue description, but may help for some people, as if we miss this step, we will lose some projects after upgrade !) 3) Upgrade to gitlab 14.1.3. 4) Go to /admin/background_migrations and wait for everything to finished (If it's not, we can't upgrade to 14.2 without migration errors) 5) Upgrade to 14.2.5

Probably step 4 could be done manually with some rake command while instance is Up.

gregmzs commented 2 years ago

I succeeded too from docker-compose runtime images 14.0.0 to 14.3.3 with the following workflow :

  1. Migration automatically to 14.1.3
  2. Into the container, run those lines : 
    bundle exec rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_build_needs,id,'[["build_id"]\, ["build_id_convert_to_bigint"]]']
bundle exec rake  gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_job_artifacts,id,'[["id"\, "job_id"]\, ["id_convert_to_bigint"\, "job_id_convert_to_bigint"]]']
bundle exec rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_stages,id,'[["id"]\, ["id_convert_to_bigint"]]']
bundle exec rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_sources_pipelines,id,'[["source_job_id"]\, ["source_job_id_convert_to_bigint"]]']
bundle exec rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,push_event_payloads,event_id,'[["event_id"]\, ["event_id_convert_to_bigint"]]'] 
bundle exec rake  gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,deployments,id,'[["deployable_id"]\, ["deployable_id_convert_to_bigint"]]']
  3. Migrate automatically to 14.2.5, and run into the container those lines: 
    bundle exec rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]']
bundle exec rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_builds,id,'[["id"\, "stage_id"]\, ["id_convert_to_bigint"\, "stage_id_convert_to_bigint"]]']
bundle exec rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_builds_metadata,id,'[["id"]\, ["id_convert_to_bigint"]]']
bundle exec rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,taggings,id,'[["id"\, "taggable_id"]\, ["id_convert_to_bigint"\, "taggable_id_convert_to_bigint"]]']
  4. Migrate automatically to 14.3.3

And now it works. Just a warning about database connection string.

blumfontein commented 2 years ago

@gregmzs There is mush easier way :)

Upgrade to 14.1.x, then go to Admin->Monitoring->Background Migrations. Take a corn and wait until everything is finished. Then upgrade to 14.2.x. Wait again. Then upgrade to 14.3.x and so on.

rzuidhof commented 2 years ago

Coming from 13.12.12 I experienced the "CopyColumnUsingBackgroundMigrationJob" issue above after going through 14.0.11 to 14.4.0 (Omnibus). According to the supported upgrade path (latest 14.0.Z -> latest 14.Y.Z) this should be possible. Will try again soon with the mentioned workaround, but it would be nice if this was smoothened in new builds.

moritzhoewer commented 2 years ago

@blumfontein thanks for that hint ;)

I had to update a very old version and scratched my head why after 13 -> 14.0.x went well, the follow-up update to latest wasn't working.... But after I made sure all background migrations were finished in 14.0.X the update to latest (which is 14.4.1 right now) worked like a charm

strarsis commented 2 years ago

@blumfontein: Thanks!

Blyrex commented 2 years ago

Thanks too!

epa64 commented 2 years ago

Hello, I can't migrate from sameersbn/gitlab:14.1.3 to sameersbn/gitlab:14.2.1. I have this error: Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active': {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"ci_job_artifacts", :column_name=>"id", :job_arguments=>[["id", "job_id"], ["id_convert_to_bigint", "job_id_convert_to_bigint"]]} gitlab |

try to run the docker command after starting the gitlab stack : docker exec -it gitlab bundle exec rake RAILS_ENV=production gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_job_artifacts,id,'[["id"\, "job_id"]\, ["id_convert_to_bigint"\, "job_id_convert_to_bigint"]]']

But the container loops and the command does not succeed.

When I launch the docker command with creation of the container after having stopped and deleted the container with the database environment I have a connection error.

I connected directly to the postgresql database instance to modify the table: gitlabhq_production=> alter table ci_job_artifacts alter column job_id TYPE bigint; itlabhq_production=> alter table ci_job_artifacts alter column id TYPE bigint;

Restarting the 14.2.1 container I still have the same error.

How to execute the command to migrate the database ?

Best regards

epa64 commented 2 years ago

I have solved the migration problem. I launched gitlab in version 14.1.3 and and ran the command: docker exec --user git -it gitlab bundle exec rake RAILS_ENV=production gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_job_artifacts,id,'[["id"\, "job_id"]\, ["id_convert_to_bigint"\, "job_id_convert_to_bigint"]]'] Check gitlab log's: logs/gitlab/application.log _Batched background migration for the given configuration is already finished: {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"ci_job_artifacts", :column_name=>"id", :job_arguments=>[["id", "job_id"], ["id_convert_to_bigint", "job_id_convert_tobigint"]]} 14.1.1

Check status: docker exec --user git -it gitlab bundle exec rake db:migrate:status ... up 20210720083432 Change application setting terms not null up 20210722110515 Revert backfill ci build trace sections for bigint conversion

I found it a bit strange that after only the id field was renamed to id_convert_to_bigint and not the job_id field. Probably due to the background task _id_convert_to_bigint | integer | | not null | 0 jobid | integer | | not null |

I stopped and destroyed the container and restarted in version 14.2.1 and it worked. The table ci_job_artifacts was modified: _id_convert_to_bigint | integer | | not null | 0 job_id_convert_tobigint | integer | | not null | 0

Check Status:

docker exec --user git -it gitlab bundle exec rake db:migrate:status up 20210809143931 Finalize job id conversion to bigint for ci job artifacts up 20210809194250 Rename tables ci build trace section up 20210811214811 Schedule copy ci builds columns to security scans up 20210812145010 Reset job token scope enabled again up 20210812171704 Create project ci feature usages up 20210813101742 Create zentao tracker data up 20210813111909 Create ci build trace metadata up 20210813195518 Schedule recalculate uuid on vulnerabilities occurrences3 up 20210816161107 Remove index containing faulty regex up 20210816183304 Schedule copy ci builds columns to security scans2 up 20210817084338 Add foreign key from ci build metadata to ci job artifacts up 20210818034001 Index historical data on recorded at up 20210818055357 Add unique commit design user mention indexes

I'm going to perform the manipulation in production and continue the migration to the last version

Best regards

aboccag commented 2 years ago

@gregmzs There is mush easier way :)

Upgrade to 14.1.x, then go to Admin->Monitoring->Background Migrations. Take a corn and wait until everything is finished. Then upgrade to 14.2.x. Wait again. Then upgrade to 14.3.x and so on.

thats the way!

MaxiReglisse commented 2 years ago

The task was longer for me since I had to launch an update from version 11.8.1... Thanks to everyone's good advice, I used the following script. You must first have a docker-compose.template.yml in which: 

image: redis:6.2
image: sameersbn/postgresql:12-20200524
image: sameersbn/gitlab:GITLAB_VERSION

GITLAB_VERSION will be replaced at run time by a specific gitlab version, from the gitlab_version variable. This script should be executed 3 times, uncommenting the chose line.

#!/bin/bash

# gitlab_version="11.8.1 11.11.0 12.0.0 12.1.0 12.2.0 13.6.1"
# problem at this step, I had to restart the update from 13.6.1...
# gitlab_version="13.6.1 13.12.1 13.12.4 14.0.0 14.1.0 14.1.1 14.1.1-1 14.1.3"
# go to Admin->Monitoring->Background Migrations. Take a corn and wait until everything is finished.
# gitlab_version="14.2.1 14.3.0 14.4.0"

for GL in $gitlab_version
do
    docker rm -f `docker ps -a -q` && docker volume rm `docker volume ls -q`
    cp docker-compose.template.yml docker-compose.${GL}.yml
    perl -p -i -e "s/GITLAB_VERSION/${GL}/" docker-compose.${GL}.yml
    docker-compose -f docker-compose.${GL}.yml up -d
    # wait at most 5 minutes before gitlab wakes up ... otherwise, display the logs
    count=300
    success=0
    while [[ $success -eq 0 ]] && [[ $count -gt 0 ]]
    do
        echo -n " $count"
        result=`lynx -dump http://localhost:10080/users/sign_in 2>>/dev/null | grep "Username or email"`
        if [[ $result =~ .*Username.* ]]
        then
            success=1
            echo "... well, gitlab is OK !"
        else
            sleep 1
            count=$((count-1))
        fi
    done
    if [[ $success -eq 0 ]]
    then
        docker logs docker-gitlab_gitlab_1
        exit
    fi
done

Hope it helps! It would be handy to have a command to know that the migration between 14.1.3 and 14.2.1 is complete ...

MaxIReglisse

xingxing122 commented 2 years ago

这个问题其实很简单，官网文档有说明，在升级14.3 之前需要升级到14.2 版本，我是在容器中运行的，我是这样解决的。 gitlab 升级到14.2.7 版本，进入到容器，docker exec -ti gitlab bash
执行在升级14.3 版本的提示的语句 gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,push_event_payloads,event_id,'[["event_id"]\, ["event_id_convert_to_bigint"]]']

gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_stages,id,'[["id"]\, ["id_convert_to_bigint"]]']

gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]']

执行的时候会比较慢，待执行成功之后，然后升级14.3 版本或者14.4 版本 14.5 版本即可，我最后升级到14.5.2 版本，没有任何问题，我的升级路线为 10.7->10.8.7 -> 11.11.8 -> 12.0.12 -> 12.1.17 -> 12.10.14 -> 13.0.14 -> 13.1.11 -> 13.8.8 -> 13.9.2-> 13.12.12 -> 14.0.11 -> 14.1.6->14.1.7->14.2.1->14.2.5->14.2.6->14.2.7-> 14.3.4->14.4.3->14.5.2

instantlinux commented 2 years ago

I'm stuck on attempting upgrade past 14.1.7. Scanned the above looking for any hint but don't see a workaround. I get an error:

PG::ForeignKeyViolation: ERROR:  insert or update on table "push_event_payloads" violates foreign key constraint "fk_36c74129da_tmp"
DETAIL:  Key (event_id)=(389) is not present in table "events".

Am running psql 12.5. Something apparently happened during a previous migration that left at least one (likely many more) dangling reference in the push_event_payloads table. This is the one for which there is no corresponding id in the events table. I'm tempted to just delete the dangling record and try again, but without any understanding of these tables it's beyond my ability to figure out. And there doesn't seem to be a database integrity checking / repair tool for gitlab.

commit_count               | 1
event_id                   | 389
action                     | 2
ref_type                   | 0
commit_from                | \xf4f70a37905670b31e3705585abad9b3a92643eb
commit_to                  | \xb0f81ef1eadf7dfceb1db49187e2baf7924a9d62
ref                        | SYS-193_python
commit_title               | SYS-193 add rotate, verify; improve error handling
ref_count                  | 
event_id_convert_to_bigint | 389

How do I repair this and do the upgrade?

xusy2k commented 2 years ago

@instantlinux: Same error here. After a lot of research I realized that some migrations had not been executed

:warning: The following answer it's based on my own experiences. Please, don't make copy & paste if you don't know what I'm doing. I encourage to use a fresh install and compare your own database with the fresh one using tools like https://www.postgrescompare.com/ :warning:

In case you don't have integrations table and you have services table, you can rename it

ALTER TABLE services RENAME TO integrations;

I've ran into other errors, also with migrations not applied.

I followed the @gregmzs' steps at https://github.com/sameersbn/docker-gitlab/issues/2409#issuecomment-949970785 but instead of execute migrations inside docker image, I checked which fields have not been migrated correctly:

SELECT
    table_name, column_name, concat('SELECT ',column_name,' FROM ',table_name,';') AS "sql_to_run"
FROM
    information_schema.columns
WHERE
    -- table_schema = 'YOUR_GITLAB_DATABASE' AND
    lower(column_name) LIKE('%convert_to_bigint%')
ORDER BY table_name, column_name ;

After checked them, I've been executed the sql for fix them:

UPDATE ci_job_artifacts SET id_convert_to_bigint=id WHERE id_convert_to_bigint=0;
UPDATE ci_job_artifacts SET job_id_convert_to_bigint=job_id WHERE job_id_convert_to_bigint=0;
UPDATE ci_stages SET id_convert_to_bigint=id WHERE id_convert_to_bigint=0;
UPDATE push_event_payloads SET event_id_convert_to_bigint=event_id WHERE event_id_convert_to_bigint=0;
UPDATE events SET id_convert_to_bigint=id WHERE id_convert_to_bigint=0;
UPDATE ci_builds SET id_convert_to_bigint=id WHERE id_convert_to_bigint=0;
UPDATE ci_builds SET stage_id_convert_to_bigint=stage_id WHERE stage_id_convert_to_bigint=0;
UPDATE ci_builds_metadata SET id_convert_to_bigint=id WHERE id_convert_to_bigint=0;
UPDATE ci_builds_metadata SET build_id_convert_to_bigint=build_id WHERE build_id_convert_to_bigint=0;
UPDATE taggings SET id_convert_to_bigint=id WHERE id_convert_to_bigint=0;
UPDATE taggings SET taggable_id_convert_to_bigint=taggable_id WHERE taggable_id_convert_to_bigint=0;
UPDATE ci_build_trace_sections SET build_id_convert_to_bigint=build_id WHERE build_id_convert_to_bigint=0;

Hope this tips can help

instantlinux commented 2 years ago

Thanks, the nature of my db corruption is very different. If I do this query:

select * from push_event_payloads where event_id not in (select id from events);

I can see there are 89 rows in push_event_payloads that refer to events that do not exist anymore in the events table. I do not know the purpose of these tables, I have no idea whether I need to somehow restore the data (which, as far as I know, has been gone for a very long time) or can just discard those 89 payload records.

This system needs an integrity-checker that offers recommendations from the people who actually know what all 430+ database tables are for, if any inconsistencies are found prior to an upgrade.

instantlinux commented 2 years ago

I watched the table for a day and now have this observation: records older than 3 calendar years in the events table get removed by a background process. Anyone with a gitlab instance running longer than that is going to get bitten by this problem if they are currently running any version prior to 14.2.

instantlinux commented 2 years ago

I reported my problem to GitLab as issue #353266.

ananiasfilho commented 2 years ago

@gregmzs There is mush easier way :) Upgrade to 14.1.x, then go to Admin->Monitoring->Background Migrations. Take a corn and wait until everything is finished. Then upgrade to 14.2.x. Wait again. Then upgrade to 14.3.x and so on.

thats the way!

I run these steps today from Gitlab 13.x to latest. But, on my procedures, i need run as follow: apt install gitlab-ce=13.12.15-ce.0 apt install gitlab-ce=14.0.12-ce.0 From here, need upgrade to minnor to major release. apt install gitlab-ce=14.1.8-ce.0 apt install gitlab-ce=14.2.0-ce.0 apt install gitlab-ce=14.2.7-ce.0 apt install gitlab-ce=14.3.0-ce.0 apt install gitlab-ce=14.3.6-ce.0 apt install gitlab-ce=14.4.0-ce.0 apt install gitlab-ce=14.4.5-ce.0 ... now im on latest 14.8.0-ce.0 When try upgrade to 14.8.1-ce.0 or 14.8.2.0 a error ocurred: Preparing to unpack .../gitlab-ce_14.8.2-ce.0_amd64.deb ...

gitlab preinstall: Checking for unmigrated data on legacy storage
gitlab preinstall: Automatically backing up only the GitLab SQL database (excluding everything else!)
rake aborted!
Errno::ENOENT: No such file or directory @ rb_check_realpath_internal - /var/opt/gitlab/gitlab-rails/shared/registry
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/files.rb:16:in `realpath'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/files.rb:16:in `initialize'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/registry.rb:10:in `initialize'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:29:in `new'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:29:in `initialize'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:12:in `new'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:12:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:backup:create
(See full trace by running task with --trace)
gitlab preinstall: 
gitlab preinstall: Database backup failed! If you want to skip this backup, run the following command and try again:
gitlab preinstall: 
gitlab preinstall:  sudo touch /etc/gitlab/skip-auto-backup
gitlab preinstall: 
dpkg: error processing archive /var/cache/apt/archives/gitlab-ce_14.8.2-ce.0_amd64.deb (--unpack):
 new gitlab-ce package pre-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 /var/cache/apt/archives/gitlab-ce_14.8.2-ce.0_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

To solve this error, i need set registry_enable to false in /etc/gitlab/gitlab.rb gitlab_rails[‘registry_enabled’] = false

ayunwSky commented 2 years ago

Hello @gregmzs @strarsis @Blyrex ,I'am upgrade from 14.1.3 to 14.3.x. Now,my gitlab version is upgrade to 14.2.5,but when I want to upgrade to 14.3.0,i got error, When I go to Admin->Monitoring->Background Migrations,ther are have only 14 finished jobs here image

How show I do next?

king4sun commented 2 years ago

bundle exec rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,push_event_payloads,event_id,'[["event_id"]\, ["event_id_convert_to_bigint"]]'] RAILS_ENV=production

this works for me.

bradc6 commented 1 year ago

I succeeded too from docker-compose runtime images 14.0.0 to 14.3.3 with the following workflow :

1. Migration automatically to 14.1.3

2. Into the container, run those lines :
bundle exec rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_build_needs,id,'[["build_id"]\, ["build_id_convert_to_bigint"]]']
bundle exec rake  gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_job_artifacts,id,'[["id"\, "job_id"]\, ["id_convert_to_bigint"\, "job_id_convert_to_bigint"]]']
bundle exec rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_stages,id,'[["id"]\, ["id_convert_to_bigint"]]']
bundle exec rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_sources_pipelines,id,'[["source_job_id"]\, ["source_job_id_convert_to_bigint"]]']
bundle exec rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,push_event_payloads,event_id,'[["event_id"]\, ["event_id_convert_to_bigint"]]'] 
bundle exec rake  gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,deployments,id,'[["deployable_id"]\, ["deployable_id_convert_to_bigint"]]']
3. Migrate automatically to 14.2.5, and run into the container those lines:
bundle exec rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]']
bundle exec rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_builds,id,'[["id"\, "stage_id"]\, ["id_convert_to_bigint"\, "stage_id_convert_to_bigint"]]']
bundle exec rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_builds_metadata,id,'[["id"]\, ["id_convert_to_bigint"]]']
bundle exec rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,taggings,id,'[["id"\, "taggable_id"]\, ["id_convert_to_bigint"\, "taggable_id_convert_to_bigint"]]']
4. Migrate automatically to 14.3.3

And now it works. Just a warning about database connection string.

Waiting didn't help me but @gregmzs solution did! TY :D

For those with omnibus installs 

#Gitlab 14.1.3
gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_build_needs,id,'[["build_id"]\, ["build_id_convert_to_bigint"]]']
gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_job_artifacts,id,'[["id"\, "job_id"]\, ["id_convert_to_bigint"\, "job_id_convert_to_bigint"]]']
gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_stages,id,'[["id"]\, ["id_convert_to_bigint"]]']
gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_sources_pipelines,id,'[["source_job_id"]\, ["source_job_id_convert_to_bigint"]]']
gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,push_event_payloads,event_id,'[["event_id"]\, ["event_id_convert_to_bigint"]]'] 
gitlab-rake  gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,deployments,id,'[["deployable_id"]\, ["deployable_id_convert_to_bigint"]]']

#Gitlab 14.2.5
gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]']
gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_builds,id,'[["id"\, "stage_id"]\, ["id_convert_to_bigint"\, "stage_id_convert_to_bigint"]]']
gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_builds_metadata,id,'[["id"]\, ["id_convert_to_bigint"]]']
gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,taggings,id,'[["id"\, "taggable_id"]\, ["id_convert_to_bigint"\, "taggable_id_convert_to_bigint"]]']