search

sameersbn / docker-gitlab

Dockerized GitLab
http://www.damagehead.com/docker-gitlab/
MIT License
7.9k stars 2.14k forks source link

Migration fails in 10.4.1 #1492

Open mikehaertl opened 6 years ago

mikehaertl commented 6 years ago

I just tried to upgrade to 10.4.1 from 10.3.2 but the container was not started successfully.

It always stopped with an error after "Migrating database..." and then restarted the same process over and over.

The full error message was:

Jan 29 17:37:40 noc docker/gitlab[853]: Migrating database...
Jan 29 17:38:17 noc docker/gitlab[853]: rake aborted!
Jan 29 17:38:17 noc docker/gitlab[853]: ActiveRecord::StatementInvalid: Mysql2::Error: Lost connection to MySQL server during query: SELECT  `application_settings`.* FROM `application_settings`  ORDER BY `application_settings`.`id` DESC LIMIT 1
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/mysql2-0.4.10/lib/mysql2/client.rb:120:in `_query'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/mysql2-0.4.10/lib/mysql2/client.rb:120:in `block in query'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/mysql2-0.4.10/lib/mysql2/client.rb:119:in `handle_interrupt'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/mysql2-0.4.10/lib/mysql2/client.rb:119:in `query'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:317:in `block in execute'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract_adapter.rb:484:in `block in log'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract_adapter.rb:478:in `log'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:317:in `execute'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/mysql2_adapter.rb:217:in `execute'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/mysql2_adapter.rb:221:in `exec_query'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/database_statements.rb:356:in `select'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/database_statements.rb:32:in `select_all'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/query_cache.rb:70:in `select_all'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/querying.rb:39:in `find_by_sql'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation.rb:639:in `exec_queries'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation.rb:515:in `load'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation.rb:243:in `to_a'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation/finder_methods.rb:511:in `find_last'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation/finder_methods.rb:160:in `last'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/querying.rb:3:in `last'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/app/models/application_setting.rb:238:in `rescue in current'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/app/models/application_setting.rb:227:in `current'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/lib/gitlab/current_settings.rb:38:in `uncached_application_settings'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/lib/gitlab/current_settings.rb:24:in `ensure_application_settings!'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/lib/gitlab/current_settings.rb:9:in `current_application_settings'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/lib/gitlab/metrics/influx_db.rb:22:in `settings'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/lib/gitlab/metrics/influx_db.rb:11:in `influx_metrics_enabled?'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/lib/gitlab/metrics.rb:7:in `enabled?'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/config/initializers/8_metrics.rb:121:in `<top (required)>'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:268:in `load'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:268:in `block in load'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:240:in `load_dependency'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:268:in `load'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/engine.rb:652:in `block in load_config_initializer'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/notifications.rb:166:in `instrument'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/engine.rb:651:in `load_config_initializer'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/engine.rb:616:in `block (2 levels) in <class:Engine>'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/engine.rb:615:in `each'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/engine.rb:615:in `block in <class:Engine>'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/initializable.rb:30:in `instance_exec'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/initializable.rb:30:in `run'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/initializable.rb:55:in `block in run_initializers'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/initializable.rb:44:in `each'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/initializable.rb:44:in `tsort_each_child'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/initializable.rb:54:in `run_initializers'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/application.rb:352:in `initialize!'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/config/environment.rb:5:in `<top (required)>'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:274:in `require'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:274:in `block in require'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:240:in `load_dependency'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:274:in `require'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/application.rb:328:in `require_environment!'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/application.rb:457:in `block in run_tasks_blocks'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Jan 29 17:38:17 noc docker/gitlab[853]: 
Jan 29 17:38:17 noc docker/gitlab[853]: Caused by:
Jan 29 17:38:17 noc docker/gitlab[853]: Mysql2::Error: Lost connection to MySQL server during query
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/mysql2-0.4.10/lib/mysql2/client.rb:120:in `_query'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/mysql2-0.4.10/lib/mysql2/client.rb:120:in `block in query'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/mysql2-0.4.10/lib/mysql2/client.rb:119:in `handle_interrupt'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/mysql2-0.4.10/lib/mysql2/client.rb:119:in `query'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:317:in `block in execute'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract_adapter.rb:484:in `block in log'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract_adapter.rb:478:in `log'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:317:in `execute'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/mysql2_adapter.rb:217:in `execute'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/mysql2_adapter.rb:221:in `exec_query'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/database_statements.rb:356:in `select'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/database_statements.rb:32:in `select_all'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/query_cache.rb:70:in `select_all'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/querying.rb:39:in `find_by_sql'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation.rb:639:in `exec_queries'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation.rb:515:in `load'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation.rb:243:in `to_a'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation/finder_methods.rb:511:in `find_last'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation/finder_methods.rb:160:in `last'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/querying.rb:3:in `last'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/app/models/application_setting.rb:238:in `rescue in current'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/app/models/application_setting.rb:227:in `current'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/lib/gitlab/current_settings.rb:38:in `uncached_application_settings'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/lib/gitlab/current_settings.rb:24:in `ensure_application_settings!'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/lib/gitlab/current_settings.rb:9:in `current_application_settings'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/lib/gitlab/metrics/influx_db.rb:22:in `settings'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/lib/gitlab/metrics/influx_db.rb:11:in `influx_metrics_enabled?'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/lib/gitlab/metrics.rb:7:in `enabled?'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/config/initializers/8_metrics.rb:121:in `<top (required)>'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:268:in `load'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:268:in `block in load'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:240:in `load_dependency'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:268:in `load'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/engine.rb:652:in `block in load_config_initializer'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/notifications.rb:166:in `instrument'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/engine.rb:651:in `load_config_initializer'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/engine.rb:616:in `block (2 levels) in <class:Engine>'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/engine.rb:615:in `each'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/engine.rb:615:in `block in <class:Engine>'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/initializable.rb:30:in `instance_exec'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/initializable.rb:30:in `run'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/initializable.rb:55:in `block in run_initializers'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/initializable.rb:44:in `each'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/initializable.rb:44:in `tsort_each_child'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/initializable.rb:54:in `run_initializers'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/application.rb:352:in `initialize!'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/config/environment.rb:5:in `<top (required)>'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:274:in `require'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:274:in `block in require'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:240:in `load_dependency'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:274:in `require'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/application.rb:328:in `require_environment!'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/application.rb:457:in `block in run_tasks_blocks'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Jan 29 17:38:17 noc docker/gitlab[853]: 
Jan 29 17:38:17 noc docker/gitlab[853]: Caused by:
Jan 29 17:38:17 noc docker/gitlab[853]: missing settings
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/app/models/application_setting.rb:233:in `block (2 levels) in current'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/app/models/application_setting.rb:231:in `tap'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/app/models/application_setting.rb:231:in `block in current'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:299:in `block in fetch'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:585:in `block in save_block_result_to_cache'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/cache.rb:299:in `fetch'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/app/models/application_setting.rb:230:in `current'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/lib/gitlab/current_settings.rb:38:in `uncached_application_settings'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/lib/gitlab/current_settings.rb:24:in `ensure_application_settings!'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/lib/gitlab/current_settings.rb:9:in `current_application_settings'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/lib/gitlab/metrics/influx_db.rb:22:in `settings'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/lib/gitlab/metrics/influx_db.rb:11:in `influx_metrics_enabled?'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/lib/gitlab/metrics.rb:7:in `enabled?'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/config/initializers/8_metrics.rb:121:in `<top (required)>'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:268:in `load'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:268:in `block in load'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:240:in `load_dependency'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:268:in `load'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/engine.rb:652:in `block in load_config_initializer'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/notifications.rb:166:in `instrument'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/engine.rb:651:in `load_config_initializer'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/engine.rb:616:in `block (2 levels) in <class:Engine>'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/engine.rb:615:in `each'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/engine.rb:615:in `block in <class:Engine>'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/initializable.rb:30:in `instance_exec'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/initializable.rb:30:in `run'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/initializable.rb:55:in `block in run_initializers'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/initializable.rb:44:in `each'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/initializable.rb:44:in `tsort_each_child'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/initializable.rb:54:in `run_initializers'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/application.rb:352:in `initialize!'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/config/environment.rb:5:in `<top (required)>'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:274:in `require'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:274:in `block in require'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:240:in `load_dependency'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:274:in `require'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/application.rb:328:in `require_environment!'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/application.rb:457:in `block in run_tasks_blocks'
Jan 29 17:38:17 noc docker/gitlab[853]: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Jan 29 17:38:17 noc docker/gitlab[853]: Tasks: TOP => db:migrate => environment
Jan 29 17:38:17 noc docker/gitlab[853]: (See full trace by running task with --trace)
maltzsama commented 6 years ago

I had same error here when I trying to update from previos versions. But I think you need update your privileges:

Your database user is not allowed to create, drop, or execute triggers on the
table web_hooks.

If you are using PostgreSQL you can solve this by logging in to the GitLab
database (gitlabhq_galileo) using a super user and running:

    ALTER gitlab WITH SUPERUSER

For MySQL you instead need to run:

    GRANT ALL PRIVILEGES ON *.* TO gitlab@'%'

Now its working to me

mikehaertl commented 6 years ago

@maltzsama Hmm, thanks. But my gitlab user in MySQL already has all privileges for the gitlab database:

SHOW GRANTS for gitlab;

Result:

GRANT USAGE ON *.* TO 'gitlab'@'%'
GRANT ALL PRIVILEGES ON `gitlab`.* TO 'gitlab'@'%'

Anything else I could try?

maltzsama commented 6 years ago

I'm not sure. I tried to do my update based on 8.x, and after based on 9.x, and my tables was configured like that:

GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, CREATE TEMPORARY TABLES, DROP, INDEX, ALTER, LOCK TABLES ON `gitlabhq_galileo`.* TO 'gitlab'@'%';

But when I tried to update I had the same error as you. So I reapply the privileges and than that's worked for me. The error log on my docker logs gitlab showed this information:

/home/git/gitlab/lib/gitlab/database/migration_helpers.rb:752:in `check_trigger_permissions!'
/home/git/gitlab/lib/gitlab/database/migration_helpers.rb:373:in `rename_column_concurrently'
/home/git/gitlab/db/migrate/20170511082759_rename_web_hooks_build_events_to_job_events.rb:12:in `up'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/migration.rb:611:in `exec_migration'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/migration.rb:592:in `block (2 levels) in migrate'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/migration.rb:591:in `block in migrate'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:292:in `with_connection'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/migration.rb:590:in `migrate'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/migration.rb:768:in `migrate'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/migration.rb:1023:in `block in execute_migration_in_transaction'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/migration.rb:1071:in `ddl_transaction'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/migration.rb:1022:in `execute_migration_in_transaction'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/migration.rb:984:in `block in migrate'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/migration.rb:980:in `each'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/migration.rb:980:in `migrate'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/migration.rb:823:in `up'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/migration.rb:801:in `migrate'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/tasks/database_tasks.rb:139:in `migrate'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/railties/databases.rake:44:in `block (2 levels) in <top (required)>'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'

Caused by:
Your database user is not allowed to create, drop, or execute triggers on the
table web_hooks.

If you are using PostgreSQL you can solve this by logging in to the GitLab
database (gitlabhq_galileo) using a super user and running:

    ALTER gitlab WITH SUPERUSER

For MySQL you instead need to run:

    GRANT ALL PRIVILEGES ON *.* TO gitlab@'%'

Both queries will grant the user super user permissions, ensuring you don't run
into similar problems in the future (e.g. when new tables are created).
mikehaertl commented 6 years ago

Ok, I think I found something. It seems that ALL PRIVILEGES is not enough to create triggers in MySQL in some configurations. The user either needs SUPER privileges or you have to set log_bin_trust_function_creators = 1.

GitLab also has this note in their upgrade guide:

https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/update/10.3-to-10.4.md#10-update-mysql-permissions

I will try to set log_bin_trust_function_creators = 1 later and report back if this helped.

Sidenote: It's weird that the GitLab guys have the same upgrade instructions over and over again. E.g. the note above is from 10.3 -> 10.4 but it's also contained in 10.2 -> 10.3 and older upgrade guides. Why all that noise? One of the reasons why I like sameer's image so much. ;)

mikehaertl commented 6 years ago

I don't think, my problem is related to permissions as my error looks a little different.

I've created an issue with GitLab as I now have even more problems: I can't even create a backup with 10.3.2 anymore.

https://gitlab.com/gitlab-org/gitlab-ce/issues/42590

mikehaertl commented 6 years ago

I've done an intermediate upgrade to 10.3.6. Now at least backups work again.

Upgrading to 10.4.2 still fails with the same error. I've also tried to set log_bin_trust_function_creators but to no avail.

ld0574 commented 6 years ago

the same issue here: ActiveRecord::StatementInvalid: Mysql2::Error: Lost connection to MySQL server during query: SELECT application_settings.* FROM application_settings ORDER BY application_settings.id DESC LIMIT 1

mikehaertl commented 6 years ago

I've also added the MySQL query log to the Gitlab issue above. But it seems, they are not interested in fixing MySQL related bugs, so I repeat it here. Maybe someone has an idea. The last query is issued twice, maybe that's a sign.

I found some pages that mention reaping_frequency as possible cause. But this is already disabled in GitLab for some time now AFAIK. So it's probably something else.

2018-01-31T14:36:40.130682Z  345 Connect        gitlab@192.168.0.8 on gitlab using TCP/IP
2018-01-31T14:36:40.142067Z  345 Query  SET NAMES utf8 COLLATE utf8_general_ci,  @@SESSION.sql_auto_is_null = 0, @@SESSION.wait_ti
meout = 2147483, @@SESSION.sql_mode = 'STRICT_ALL_TABLES'
2018-01-31T14:36:40.145440Z  345 Query  SHOW TABLES LIKE 'application_settings'
2018-01-31T14:36:40.149082Z  345 Query  SHOW FULL FIELDS FROM `application_settings`
2018-01-31T14:36:40.172786Z  345 Query  SHOW TABLES
2018-01-31T14:36:40.174938Z  345 Query  SHOW CREATE TABLE `application_settings`
2018-01-31T14:36:40.308565Z  345 Query  SELECT  `application_settings`.* FROM `application_settings`  ORDER BY `application_settin
gs`.`id` DESC LIMIT 1
2018-01-31T14:36:40.328810Z  345 Query  SELECT  `application_settings`.* FROM `application_settings`  ORDER BY `application_settin
gs`.`id` DESC LIMIT 1
mikehaertl commented 6 years ago

I wonder if it could have to do with the upgrade to Ubuntu Xenial as base image since 10.4.0. Maybe some Ruby related stuff fundamentally changed there?

https://github.com/sameersbn/docker-gitlab/blob/master/Changelog.md

@solidnerd Do you maybe have an idea what could cause the issues we have?

mikehaertl commented 6 years ago

@ld0574 I wonder if it has to do with MySQL version. Which one do you have? Mine is 5.7.11.

ld0574 commented 6 years ago

@mikehaertl Mine is 5.7.21, Maybe mysql connector inside was out-dated

solidnerd commented 6 years ago

@mikehaertl I have at no clue at the moment. I personally don't use mysql as database backend. But I know that @splattael is using mysql as database backend.

@splattael Did the last update work for you ? And what is your mysql server version ?

splattael commented 6 years ago

@solidnerd I experienced some collation problems with MySQL (also 5.7.21) during the upgrade, but I think this was my own fault. Creating a backup works like a charm. No problems at all :/

ld0574 commented 6 years ago

I solved the problem by modify mysql's config file: max_connections. @mikehaertl Make sure max_connections > Number of Rails Process * Size of Connection Pool

mikehaertl commented 6 years ago

@ld0574 Wow, this seems to indeed have fixed it. Thanks! I'm on 10.5.1 now. Without this change, the upgrade still failed.

But it makes absolutely no sense to me. I've increased the max_connections from default 100 to 500. I've checked the number of connections before and after with show status like '%onn%';. The Max_used_connections is somewhere in the <10 range. So how could this possibly be the problem?

mikehaertl commented 6 years ago

Has anyone tried the latest 11.0.1? Gitlab now does not work at all anymore for me. I get similar errors like above. See #1652. Does anyone else have this?

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had any activity for the last 60 days. It will be closed if no further activity occurs during the next 7 days. Thank you for your contributions.