search

cloudfoundry / capi-release

Bosh Release for Cloud Controller and friends
Apache License 2.0
24 stars 101 forks source link

Stack Checker causes CC not to start if upgrading from an older CAPI release #463

Closed Samze closed 2 weeks ago

Samze commented 3 weeks ago

Issue

Logic to check for deprecated stacks was added and is currently being run before migrations, see here. This runs before migrations so that if someone is using a deprecated stack, the checker fails before migrations are run so that you are not left in a state where new migrations have run but you are running old CC code.

However, this Stack Checker rake task loads all models, https://github.com/cloudfoundry/cloud_controller_ng/blob/main/lib/tasks/stack_check.rake . This causes issues when migrations have not run yet and there is a migration that is required for the model load to complete. For example adding a new table, which was done recently https://github.com/cloudfoundry/cloud_controller_ng/blob/main/db/migrations/20240514113500_create_cnb_lifecycle_data.rb . This will cause CC to fail to start as the Stack checker rake task will fail with something like in pre-start:

[2024-08-27 15:24:42+0000] + bundle exec rake stacks:stack_check
[2024-08-27 15:24:47+0000] rake aborted!
[2024-08-27 15:24:47+0000] Sequel::DatabaseError: Mysql2::Error: Table 'cloud_controller.cnb_lifecycle_data' doesn't exist (Sequel::DatabaseError)
[2024-08-27 15:24:47+0000] /var/vcap/data/packages/cloud_controller_ng/a761d212d4175fe3f27777c61f0de6428e52f8b4/gem_home/ruby/3.2.0/gems/mysql2-0.5.6/lib/mysql2/client.rb:151:in `_query'
[2024-08-27 15:24:47+0000] /var/vcap/data/packages/cloud_controller_ng/a761d212d4175fe3f27777c61f0de6428e52f8b4/gem_home/ruby/3.2.0/gems/mysql2-0.5.6/lib/mysql2/client.rb:151:in `block in query'
[2024-08-27 15:24:47+0000] /var/vcap/data/packages/cloud_controller_ng/a761d212d4175fe3f27777c61f0de6428e52f8b4/gem_home/ruby/3.2.0/gems/mysql2-0.5.6/lib/mysql2/client.rb:150:in `handle_interrupt'
[2024-08-27 15:24:47+0000] /var/vcap/data/packages/cloud_controller_ng/a761d212d4175fe3f27777c61f0de6428e52f8b4/gem_home/ruby/3.2.0/gems/mysql2-0.5.6/lib/mysql2/client.rb:150:in `query'
[2024-08-27 15:24:47+0000] /var/vcap/data/packages/cloud_controller_ng/a761d212d4175fe3f27777c61f0de6428e52f8b4/gem_home/ruby/3.2.0/gems/sequel-5.83.1/lib/sequel/adapters/mysql2.rb:142:in `block in _execute'
[2024-08-27 15:24:47+0000] /var/vcap/data/packages/cloud_controller_ng/a761d212d4175fe3f27777c61f0de6428e52f8b4/gem_home/ruby/3.2.0/gems/sequel-5.83.1/lib/sequel/database/logging.rb:38:in `log_connection_yield'
[2024-08-27 15:24:48+0000] /var/vcap/data/packages/cloud_controller_ng/a761d212d4175fe3f27777c61f0de6428e52f8b4/gem_home/ruby/3.2.0/gems/honeycomb-beeline-3.1.0/lib/honeycomb/integrations/sequel.rb:17:in `log_connection_yield'
[2024-08-27 15:24:48+0000] /var/vcap/data/packages/cloud_controller_ng/a761d212d4175fe3f27777c61f0de6428e52f8b4/gem_home/ruby/3.2.0/gems/sequel-5.83.1/lib/sequel/adapters/mysql2.rb:137:in `_execute'
[2024-08-27 15:24:48+0000] /var/vcap/data/packages/cloud_controller_ng/a761d212d4175fe3f27777c61f0de6428e52f8b4/gem_home/ruby/3.2.0/gems/sequel-5.83.1/lib/sequel/adapters/utils/mysql_mysql2.rb:40:in `block in execute'
[2024-08-27 15:24:48+0000] /var/vcap/data/packages/cloud_controller_ng/a761d212d4175fe3f27777c61f0de6428e52f8b4/gem_home/ruby/3.2.0/gems/sequel-5.83.1/lib/sequel/connection_pool/threaded.rb:88:in `hold'

This issue is present in CAPI Release 1.190.0 https://github.com/cloudfoundry/capi-release/releases/tag/1.190.0 and may be hit if upgrading from an older capi release where the delta in migrations contains new tables.

Possible Fix

Do not load all models in stack checker, or better yet load no models and just use raw sql.

Samze commented 3 weeks ago

Fix here: https://github.com/cloudfoundry/cloud_controller_ng/pull/3941

Samze commented 2 weeks ago

Fixed and released in https://github.com/cloudfoundry/capi-release/releases/tag/1.191.0