unknown attribute singleton error during ManageAccountBanksBackfill migration (canvas-lms/stable 2021-10-21)

[skylar-cognella]

Summary:

We're testing canvas-lms/stable from 2021-10-21 and running into a db:migration error for ManageAccountBanksBackfill.

Steps to reproduce:

1. Upgrading from canvas-lms/stable 2020-11-30 to canvas-lms/stable 2021-10-21.
2. Running db:migrate:predeploy results in an error for ManageAccountBanksBackfill

Already tried clearing redis cache and re-running migrate with no luck. Looking forward to any help on what the issue could be. Any files we can tweak to get passed this migration error? Thanks!

Behavior:

Error returned:

==  ManageAccountBanksBackfill: migrating =====================================
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/call_super.rb:16: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation/batches.rb:201: warning: The called method `in_batches' is defined here
/var/canvas/vendor/bundle/ruby/2.7.0/gems/inst-jobs-3.0.0/lib/delayed/message_sending.rb:68: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/var/canvas/vendor/bundle/ruby/2.7.0/gems/inst-jobs-3.0.0/lib/delayed/performable_method.rb:5: warning: The called method `initialize' is defined here
rake aborted!
StandardError: An error has occurred, all later migrations canceled:

unknown attribute 'singleton' for Delayed::Backend::ActiveRecord::Job.
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activemodel-6.0.4.1/lib/active_model/attribute_assignment.rb:52:in `_assign_attribute'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activemodel-6.0.4.1/lib/active_model/attribute_assignment.rb:43:in `block in _assign_attributes'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activemodel-6.0.4.1/lib/active_model/attribute_assignment.rb:42:in `each'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activemodel-6.0.4.1/lib/active_model/attribute_assignment.rb:42:in `_assign_attributes'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/attribute_assignment.rb:21:in `_assign_attributes'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activemodel-6.0.4.1/lib/active_model/attribute_assignment.rb:35:in `assign_attributes'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/core.rb:324:in `initialize'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/aroi-0.0.7/lib/aroi/instrumentation/active_record.rb:34:in `block in initialize'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/notifications.rb:180:in `block in instrument'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/notifications.rb:180:in `instrument'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/aroi-0.0.7/lib/aroi/instrumentation/active_record.rb:33:in `initialize'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/inheritance.rb:70:in `new'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/inheritance.rb:70:in `new'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/inst-jobs-3.0.0/lib/delayed/backend/active_record.rb:38:in `create'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/inst-jobs-3.0.0/lib/delayed/backend/base.rb:89:in `enqueue'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-inst-jobs-3.2.4/lib/switchman_inst_jobs/delayed/backend/base.rb:21:in `block (2 levels) in enqueue'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-inst-jobs-3.2.4/lib/switchman_inst_jobs/guard_rail.rb:6:in `activate'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-inst-jobs-3.2.4/lib/switchman_inst_jobs/delayed/backend/base.rb:21:in `block in enqueue'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-inst-jobs-3.2.4/lib/switchman_inst_jobs/delayed/backend/base.rb:40:in `block in enqueue'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/app/models/switchman/shard.rb:596:in `block in activate'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/app/models/switchman/shard.rb:98:in `activate'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/app/models/switchman/shard.rb:595:in `activate'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-inst-jobs-3.2.4/lib/switchman_inst_jobs/delayed/backend/base.rb:39:in `enqueue'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/inst-jobs-3.0.0/lib/delayed/message_sending.rb:68:in `method_missing'
/var/canvas/app/models/role_override.rb:43:in `clear_caches'
/var/canvas/app/models/role_override.rb:38:in `clear_caches'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/callbacks.rb:428:in `block in make_lambda'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/callbacks.rb:238:in `block in halting_and_conditional'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/callbacks.rb:517:in `block in invoke_after'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/callbacks.rb:517:in `each'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/callbacks.rb:517:in `invoke_after'
/var/canvas/gems/activesupport-suspend_callbacks/lib/active_support/callbacks/suspension.rb:181:in `run_callbacks'
/var/canvas/config/initializers/active_record.rb:1747:in `block in run_callbacks'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation.rb:407:in `block in scoping'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation.rb:787:in `_scoping'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation.rb:407:in `scoping'
/var/canvas/config/initializers/active_record.rb:1747:in `run_callbacks'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/callbacks.rb:825:in `_run_save_callbacks'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/callbacks.rb:327:in `create_or_update'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/timestamp.rb:128:in `create_or_update'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/persistence.rb:503:in `save!'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/validations.rb:53:in `save!'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/transactions.rb:318:in `block in save!'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/transactions.rb:375:in `block in with_transaction_returning_status'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `block in transaction'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/abstract/transaction.rb:280:in `block in within_new_transaction'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/abstract/transaction.rb:278:in `within_new_transaction'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `transaction'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/transactions.rb:212:in `transaction'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/active_record/base.rb:43:in `block in transaction'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/active_record/relation.rb:104:in `activate'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/active_record/base.rb:38:in `transaction'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/transactions.rb:366:in `with_transaction_returning_status'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/transactions.rb:318:in `save!'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/suppressor.rb:48:in `save!'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/active_record/base.rb:117:in `block in save!'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation.rb:407:in `block in scoping'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation.rb:787:in `_scoping'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation.rb:407:in `scoping'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/active_record/base.rb:117:in `save!'
/var/canvas/lib/data_fixup/add_role_overrides_for_new_permission.rb:46:in `block (2 levels) in run'
/var/canvas/lib/data_fixup/add_role_overrides_for_new_permission.rb:39:in `each'
/var/canvas/lib/data_fixup/add_role_overrides_for_new_permission.rb:39:in `block in run'
/var/canvas/config/initializers/active_record.rb:779:in `block in find_in_batches'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation/batches.rb:238:in `block in in_batches'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation/batches.rb:222:in `loop'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation/batches.rb:222:in `in_batches'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/call_super.rb:16:in `call'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/call_super.rb:16:in `call_super'
/var/canvas/config/initializers/active_record.rb:793:in `block in in_batches'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/active_record/relation.rb:104:in `activate'
/var/canvas/config/initializers/active_record.rb:793:in `in_batches'
/var/canvas/config/initializers/active_record.rb:778:in `find_in_batches'
/var/canvas/lib/data_fixup/add_role_overrides_for_new_permission.rb:35:in `run'
/var/canvas/db/migrate/20210715175216_manage_account_banks_backfill.rb:25:in `up'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:831:in `exec_migration'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:812:in `block (2 levels) in migrate'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:811:in `block in migrate'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:471:in `with_connection'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/connection_pool_proxy.rb:15:in `with_connection'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:810:in `migrate'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-inst-jobs-3.2.4/lib/switchman_inst_jobs/active_record/migration.rb:16:in `migrate'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:624:in `migrate'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:1002:in `migrate'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:1310:in `block in execute_migration_in_transaction'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:1363:in `ddl_transaction'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:1309:in `execute_migration_in_transaction'
/var/canvas/config/initializers/active_record.rb:1611:in `execute_migration_in_transaction'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:1281:in `block in migrate_without_lock'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:1280:in `each'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:1280:in `migrate_without_lock'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:1229:in `block in migrate'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/active_record/migration.rb:51:in `block in with_advisory_lock'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/active_record/migration.rb:67:in `block in with_advisory_lock_connection'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:471:in `with_connection'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/connection_pool_proxy.rb:15:in `with_connection'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/active_record/migration.rb:67:in `with_advisory_lock_connection'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/active_record/migration.rb:47:in `with_advisory_lock'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:1229:in `migrate'
/var/canvas/lib/tasks/canvas.rake:175:in `block (3 levels) in <top (required)>'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/tasks/switchman.rake:88:in `block (6 levels) in shardify_task'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/tasks/switchman.rake:88:in `each'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/tasks/switchman.rake:88:in `block (5 levels) in shardify_task'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/database_server.rb:109:in `unguard'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/tasks/switchman.rake:87:in `block (4 levels) in shardify_task'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/app/models/switchman/shard.rb:362:in `block (2 levels) in with_each_shard'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/app/models/switchman/shard.rb:596:in `block in activate'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/app/models/switchman/shard.rb:98:in `activate'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/app/models/switchman/shard.rb:595:in `activate'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/app/models/switchman/shard.rb:360:in `block in with_each_shard'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation/delegation.rb:88:in `each'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation/delegation.rb:88:in `each'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/app/models/switchman/shard.rb:356:in `with_each_shard'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/tasks/switchman.rake:72:in `block (3 levels) in shardify_task'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/database_server.rb:109:in `unguard'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/tasks/switchman.rake:69:in `block (2 levels) in shardify_task'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/guard_rail.rb:24:in `activate'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-inst-jobs-3.2.4/lib/switchman_inst_jobs/guard_rail.rb:8:in `activate'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/tasks/switchman.rake:68:in `block in shardify_task'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/rake-13.0.3/exe/rake:27:in `<top (required)>'

Caused by:
ActiveModel::UnknownAttributeError: unknown attribute 'singleton' for Delayed::Backend::ActiveRecord::Job.
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activemodel-6.0.4.1/lib/active_model/attribute_assignment.rb:52:in `_assign_attribute'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activemodel-6.0.4.1/lib/active_model/attribute_assignment.rb:43:in `block in _assign_attributes'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activemodel-6.0.4.1/lib/active_model/attribute_assignment.rb:42:in `each'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activemodel-6.0.4.1/lib/active_model/attribute_assignment.rb:42:in `_assign_attributes'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/attribute_assignment.rb:21:in `_assign_attributes'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activemodel-6.0.4.1/lib/active_model/attribute_assignment.rb:35:in `assign_attributes'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/core.rb:324:in `initialize'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/aroi-0.0.7/lib/aroi/instrumentation/active_record.rb:34:in `block in initialize'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/notifications.rb:180:in `block in instrument'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/notifications.rb:180:in `instrument'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/aroi-0.0.7/lib/aroi/instrumentation/active_record.rb:33:in `initialize'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/inheritance.rb:70:in `new'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/inheritance.rb:70:in `new'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/inst-jobs-3.0.0/lib/delayed/backend/active_record.rb:38:in `create'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/inst-jobs-3.0.0/lib/delayed/backend/base.rb:89:in `enqueue'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-inst-jobs-3.2.4/lib/switchman_inst_jobs/delayed/backend/base.rb:21:in `block (2 levels) in enqueue'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-inst-jobs-3.2.4/lib/switchman_inst_jobs/guard_rail.rb:6:in `activate'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-inst-jobs-3.2.4/lib/switchman_inst_jobs/delayed/backend/base.rb:21:in `block in enqueue'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-inst-jobs-3.2.4/lib/switchman_inst_jobs/delayed/backend/base.rb:40:in `block in enqueue'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/app/models/switchman/shard.rb:596:in `block in activate'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/app/models/switchman/shard.rb:98:in `activate'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/app/models/switchman/shard.rb:595:in `activate'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-inst-jobs-3.2.4/lib/switchman_inst_jobs/delayed/backend/base.rb:39:in `enqueue'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/inst-jobs-3.0.0/lib/delayed/message_sending.rb:68:in `method_missing'
/var/canvas/app/models/role_override.rb:43:in `clear_caches'
/var/canvas/app/models/role_override.rb:38:in `clear_caches'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/callbacks.rb:428:in `block in make_lambda'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/callbacks.rb:238:in `block in halting_and_conditional'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/callbacks.rb:517:in `block in invoke_after'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/callbacks.rb:517:in `each'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/callbacks.rb:517:in `invoke_after'
/var/canvas/gems/activesupport-suspend_callbacks/lib/active_support/callbacks/suspension.rb:181:in `run_callbacks'
/var/canvas/config/initializers/active_record.rb:1747:in `block in run_callbacks'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation.rb:407:in `block in scoping'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation.rb:787:in `_scoping'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation.rb:407:in `scoping'
/var/canvas/config/initializers/active_record.rb:1747:in `run_callbacks'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/callbacks.rb:825:in `_run_save_callbacks'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/callbacks.rb:327:in `create_or_update'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/timestamp.rb:128:in `create_or_update'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/persistence.rb:503:in `save!'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/validations.rb:53:in `save!'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/transactions.rb:318:in `block in save!'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/transactions.rb:375:in `block in with_transaction_returning_status'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `block in transaction'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/abstract/transaction.rb:280:in `block in within_new_transaction'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/abstract/transaction.rb:278:in `within_new_transaction'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `transaction'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/transactions.rb:212:in `transaction'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/active_record/base.rb:43:in `block in transaction'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/active_record/relation.rb:104:in `activate'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/active_record/base.rb:38:in `transaction'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/transactions.rb:366:in `with_transaction_returning_status'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/transactions.rb:318:in `save!'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/suppressor.rb:48:in `save!'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/active_record/base.rb:117:in `block in save!'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation.rb:407:in `block in scoping'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation.rb:787:in `_scoping'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation.rb:407:in `scoping'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/active_record/base.rb:117:in `save!'
/var/canvas/lib/data_fixup/add_role_overrides_for_new_permission.rb:46:in `block (2 levels) in run'
/var/canvas/lib/data_fixup/add_role_overrides_for_new_permission.rb:39:in `each'
/var/canvas/lib/data_fixup/add_role_overrides_for_new_permission.rb:39:in `block in run'
/var/canvas/config/initializers/active_record.rb:779:in `block in find_in_batches'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation/batches.rb:238:in `block in in_batches'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation/batches.rb:222:in `loop'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation/batches.rb:222:in `in_batches'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/call_super.rb:16:in `call'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/call_super.rb:16:in `call_super'
/var/canvas/config/initializers/active_record.rb:793:in `block in in_batches'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/active_record/relation.rb:104:in `activate'
/var/canvas/config/initializers/active_record.rb:793:in `in_batches'
/var/canvas/config/initializers/active_record.rb:778:in `find_in_batches'
/var/canvas/lib/data_fixup/add_role_overrides_for_new_permission.rb:35:in `run'
/var/canvas/db/migrate/20210715175216_manage_account_banks_backfill.rb:25:in `up'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:831:in `exec_migration'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:812:in `block (2 levels) in migrate'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:811:in `block in migrate'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:471:in `with_connection'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/connection_pool_proxy.rb:15:in `with_connection'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:810:in `migrate'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-inst-jobs-3.2.4/lib/switchman_inst_jobs/active_record/migration.rb:16:in `migrate'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:624:in `migrate'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:1002:in `migrate'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:1310:in `block in execute_migration_in_transaction'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:1363:in `ddl_transaction'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:1309:in `execute_migration_in_transaction'
/var/canvas/config/initializers/active_record.rb:1611:in `execute_migration_in_transaction'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:1281:in `block in migrate_without_lock'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:1280:in `each'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:1280:in `migrate_without_lock'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:1229:in `block in migrate'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/active_record/migration.rb:51:in `block in with_advisory_lock'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/active_record/migration.rb:67:in `block in with_advisory_lock_connection'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:471:in `with_connection'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/connection_pool_proxy.rb:15:in `with_connection'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/active_record/migration.rb:67:in `with_advisory_lock_connection'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/active_record/migration.rb:47:in `with_advisory_lock'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/migration.rb:1229:in `migrate'
/var/canvas/lib/tasks/canvas.rake:175:in `block (3 levels) in <top (required)>'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/tasks/switchman.rake:88:in `block (6 levels) in shardify_task'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/tasks/switchman.rake:88:in `each'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/tasks/switchman.rake:88:in `block (5 levels) in shardify_task'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/database_server.rb:109:in `unguard'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/tasks/switchman.rake:87:in `block (4 levels) in shardify_task'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/app/models/switchman/shard.rb:362:in `block (2 levels) in with_each_shard'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/app/models/switchman/shard.rb:596:in `block in activate'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/app/models/switchman/shard.rb:98:in `activate'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/app/models/switchman/shard.rb:595:in `activate'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/app/models/switchman/shard.rb:360:in `block in with_each_shard'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation/delegation.rb:88:in `each'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.4.1/lib/active_record/relation/delegation.rb:88:in `each'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/app/models/switchman/shard.rb:356:in `with_each_shard'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/tasks/switchman.rake:72:in `block (3 levels) in shardify_task'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/database_server.rb:109:in `unguard'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/tasks/switchman.rake:69:in `block (2 levels) in shardify_task'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/switchman/guard_rail.rb:24:in `activate'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-inst-jobs-3.2.4/lib/switchman_inst_jobs/guard_rail.rb:8:in `activate'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/switchman-2.2.2/lib/tasks/switchman.rake:68:in `block in shardify_task'
/var/canvas/vendor/bundle/ruby/2.7.0/gems/rake-13.0.3/exe/rake:27:in `<top (required)>'
Tasks: TOP => db:migrate:predeploy
(See full trace by running task with --trace)

[skylar-cognella]

While I haven't heard back, I was able to workaround this by moving db/migrate/20210715175216_manage_account_banks_backfill.rb out of that folder, running db:migrate:predeploy and letting it complete all the predeploy migrations, moving it back in, and running db:migrate:predeploy again. So something with the order of the migration scripts was causing this to fail.

Hopefully by running it last (out of order) that doesn't cause any issues - haven't seen any issues so far though!

[breiter]

I also ran into the same problem. I think that perhaps db/migrate/20210715175216_manage_account_banks_backfill.rb depends on the later migration db/migrate/20210812210129_add_singleton_column.rb.

[nathanielb]

Yes, exec rake db:migrate:up VERSION=20210812210129 then running predeploy again fixed this issue for me.