instructure / canvas-lms

The open LMS by Instructure, Inc.
https://github.com/instructure/canvas-lms/wiki
GNU Affero General Public License v3.0
5.67k stars 2.5k forks source link

delayed notification processing failed: PG::UndefinedTable: ERROR: relation does not exist #1954

Closed amg-web closed 2 years ago

amg-web commented 3 years ago

Delayed jobs stop working with following error. Need help to resolve it .

[canvas.d.com:4999 11349066] delayed notification processing failed: PG::UndefinedTable: ERROR:  relation "public.messages_2021_42" does not exist
LINE 1: INSERT INTO "public"."messages_2021_42" ("to", "subject", "b...
                    ^

/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/postgresql_adapter.rb:675:in `exec_params'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/postgresql_adapter.rb:675:in `block (2 levels) in exec_no_cache'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.4.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.4.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.4.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/postgresql_adapter.rb:674:in `block in exec_no_cache'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/abstract_adapter.rb:728:in `block (2 levels) in log'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/abstract_adapter.rb:727:in `block in log'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.4.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/usr/lib/ruby/2.6.0/delegate.rb:83:in `method_missing'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/switchman-2.2.1/lib/switchman/sharded_instrumenter.rb:22:in `instrument'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/abstract_adapter.rb:718:in `log'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/switchman-2.2.1/lib/switchman/active_record/abstract_adapter.rb:39:in `log'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/postgresql_adapter.rb:673:in `exec_no_cache'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/postgresql_adapter.rb:654:in `execute_and_clear'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:98:in `exec_query'
/home/canvas/public_html/config/initializers/postgresql_adapter.rb:341:in `exec_query'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/abstract/database_statements.rb:135:in `exec_insert'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:132:in `exec_insert'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/abstract/database_statements.rb:166:in `insert'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/abstract/query_cache.rb:22:in `insert'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/persistence.rb:375:in `_insert_record'
/home/canvas/public_html/gems/canvas_partman/lib/canvas_partman/concerns/partitioned.rb:136:in `_insert_record'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/persistence.rb:932:in `_create_record'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/counter_cache.rb:166:in `_create_record'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/locking/optimistic.rb:79:in `_create_record'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/attribute_methods/dirty.rb:211:in `_create_record'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/callbacks.rb:331:in `block in _create_record'
/home/canvas/public_html/gems/activesupport-suspend_callbacks/lib/active_support/callbacks/suspension.rb:179:in `run_callbacks'
/home/canvas/public_html/config/initializers/active_record.rb:1751:in `block in run_callbacks'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/relation.rb:407:in `block in scoping'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/relation.rb:787:in `_scoping'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/relation.rb:407:in `scoping'
/home/canvas/public_html/config/initializers/active_record.rb:1751:in `run_callbacks'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.4.1/lib/active_support/callbacks.rb:825:in `_run_create_callbacks'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/callbacks.rb:331:in `_create_record'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/timestamp.rb:110:in `_create_record'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/persistence.rb:905:in `create_or_update'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/callbacks.rb:327:in `block in create_or_update'
/home/canvas/public_html/gems/activesupport-suspend_callbacks/lib/active_support/callbacks/suspension.rb:156:in `block in run_callbacks'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/autosave_association.rb:366:in `around_save_collection_association'
/home/canvas/public_html/gems/activesupport-suspend_callbacks/lib/active_support/callbacks/suspension.rb:165:in `block in run_callbacks'
/home/canvas/public_html/gems/activesupport-suspend_callbacks/lib/active_support/callbacks/suspension.rb:183:in `run_callbacks'
/home/canvas/public_html/config/initializers/active_record.rb:1751:in `block in run_callbacks'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/relation.rb:407:in `block in scoping'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/relation.rb:787:in `_scoping'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/relation.rb:407:in `scoping'
/home/canvas/public_html/config/initializers/active_record.rb:1751:in `run_callbacks'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.4.1/lib/active_support/callbacks.rb:825:in `_run_save_callbacks'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/callbacks.rb:327:in `create_or_update'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/timestamp.rb:128:in `create_or_update'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/persistence.rb:503:in `save!'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/validations.rb:53:in `save!'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/transactions.rb:318:in `block in save!'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/transactions.rb:375:in `block in with_transaction_returning_status'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/abstract/database_statements.rb:278:in `transaction'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/transactions.rb:212:in `transaction'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/switchman-2.2.1/lib/switchman/active_record/base.rb:43:in `block in transaction'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/switchman-2.2.1/lib/switchman/active_record/relation.rb:104:in `activate'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/switchman-2.2.1/lib/switchman/active_record/base.rb:38:in `transaction'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/transactions.rb:366:in `with_transaction_returning_status'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/transactions.rb:318:in `save!'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/suppressor.rb:48:in `save!'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/switchman-2.2.1/lib/switchman/active_record/base.rb:117:in `block in save!'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/relation.rb:407:in `block in scoping'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/relation.rb:787:in `_scoping'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/relation.rb:407:in `scoping'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/switchman-2.2.1/lib/switchman/active_record/base.rb:117:in `save!'
/home/canvas/public_html/lib/notification_message_creator.rb:193:in `block (2 levels) in dispatch_immediate_messages'
/home/canvas/public_html/lib/notification_message_creator.rb:191:in `each'
/home/canvas/public_html/lib/notification_message_creator.rb:191:in `block in dispatch_immediate_messages'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `block in transaction'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/abstract/transaction.rb:280:in `block in within_new_transaction'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/abstract/transaction.rb:278:in `within_new_transaction'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `transaction'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.4.1/lib/active_record/transactions.rb:212:in `transaction'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/switchman-2.2.1/lib/switchman/active_record/base.rb:51:in `transaction'
/home/canvas/public_html/lib/notification_message_creator.rb:188:in `dispatch_immediate_messages'
/home/canvas/public_html/lib/notification_message_creator.rb:96:in `create_message'
/home/canvas/public_html/app/models/notification.rb:208:in `create_message'
/home/canvas/public_html/app/models/delayed_notification.rb:62:in `block in process'
/home/canvas/public_html/app/models/delayed_notification.rb:100:in `block (2 levels) in iterate_to_list'
/home/canvas/public_html/app/models/delayed_notification.rb:99:in `each'
/home/canvas/public_html/app/models/delayed_notification.rb:99:in `each_slice'
/home/canvas/public_html/app/models/delayed_notification.rb:99:in `block in iterate_to_list'
/home/canvas/public_html/app/models/delayed_notification.rb:86:in `each'
/home/canvas/public_html/app/models/delayed_notification.rb:86:in `iterate_to_list'
/home/canvas/public_html/app/models/delayed_notification.rb:61:in `process'
/home/canvas/public_html/app/models/delayed_notification.rb:55:in `process'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/performable_method.rb:42:in `perform'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/backend/base.rb:282:in `block in invoke_job'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/lifecycle.rb:70:in `block in initialize'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/lifecycle.rb:75:in `execute'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/lifecycle.rb:48:in `run_callbacks'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/backend/base.rb:279:in `invoke_job'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/switchman-inst-jobs-3.1.2/lib/switchman_inst_jobs/delayed/backend/base.rb:84:in `block in invoke_job'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/switchman-2.2.1/app/models/switchman/shard.rb:596:in `block in activate'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/switchman-2.2.1/app/models/switchman/shard.rb:98:in `activate'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/switchman-2.2.1/app/models/switchman/shard.rb:595:in `activate'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/switchman-inst-jobs-3.1.2/lib/switchman_inst_jobs/delayed/backend/base.rb:84:in `invoke_job'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/worker.rb:232:in `block (2 levels) in perform'
/usr/lib/ruby/2.6.0/benchmark.rb:308:in `realtime'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/worker.rb:226:in `block in perform'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-statsd-2.1.0/lib/inst_jobs_statsd/stats/timing/perform.rb:22:in `block (2 levels) in enable_perform_timing'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-statsd-2.1.0/lib/inst_jobs_statsd/stats/timing.rb:8:in `block in report_timing'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst_statsd-2.2.0/lib/inst_statsd/statsd.rb:91:in `time'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-statsd-2.1.0/lib/inst_jobs_statsd/stats/timing.rb:8:in `report_timing'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-statsd-2.1.0/lib/inst_jobs_statsd/stats/timing/perform.rb:21:in `block in enable_perform_timing'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/lifecycle.rb:88:in `block (2 levels) in add'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-statsd-2.1.0/lib/inst_jobs_statsd/stats/timing/perform.rb:13:in `block (2 levels) in enable_batching'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst_statsd-2.2.0/lib/inst_statsd/statsd.rb:97:in `batch'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-statsd-2.1.0/lib/inst_jobs_statsd/stats/timing/perform.rb:12:in `block in enable_batching'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/lifecycle.rb:88:in `block (2 levels) in add'
/home/canvas/public_html/config/initializers/delayed_job.rb:153:in `block (2 levels) in <top (required)>'
/home/canvas/public_html/lib/temp_cache.rb:25:in `enable'
/home/canvas/public_html/config/initializers/delayed_job.rb:152:in `block in <top (required)>'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/lifecycle.rb:88:in `block (2 levels) in add'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/lifecycle.rb:70:in `block in initialize'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/lifecycle.rb:88:in `block in add'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/lifecycle.rb:88:in `block in add'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/lifecycle.rb:88:in `block in add'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/lifecycle.rb:75:in `execute'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/lifecycle.rb:48:in `run_callbacks'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/worker.rb:223:in `perform'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/worker.rb:263:in `block (2 levels) in perform_batch'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/worker.rb:298:in `block in configure_for_job'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/worker.rb:53:in `running_job'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/worker.rb:294:in `configure_for_job'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/worker.rb:262:in `block in perform_batch'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/worker.rb:259:in `each'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/worker.rb:259:in `perform_batch'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/worker.rb:230:in `block (2 levels) in perform'
/usr/lib/ruby/2.6.0/benchmark.rb:308:in `realtime'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/worker.rb:226:in `block in perform'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-statsd-2.1.0/lib/inst_jobs_statsd/stats/timing/perform.rb:22:in `block (2 levels) in enable_perform_timing'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-statsd-2.1.0/lib/inst_jobs_statsd/stats/timing.rb:8:in `block in report_timing'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst_statsd-2.2.0/lib/inst_statsd/statsd.rb:91:in `time'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-statsd-2.1.0/lib/inst_jobs_statsd/stats/timing.rb:8:in `report_timing'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-statsd-2.1.0/lib/inst_jobs_statsd/stats/timing/perform.rb:21:in `block in enable_perform_timing'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/lifecycle.rb:88:in `block (2 levels) in add'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-statsd-2.1.0/lib/inst_jobs_statsd/stats/timing/perform.rb:13:in `block (2 levels) in enable_batching'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst_statsd-2.2.0/lib/inst_statsd/statsd.rb:97:in `batch'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-statsd-2.1.0/lib/inst_jobs_statsd/stats/timing/perform.rb:12:in `block in enable_batching'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/lifecycle.rb:88:in `block (2 levels) in add'
/home/canvas/public_html/config/initializers/delayed_job.rb:153:in `block (2 levels) in <top (required)>'
/home/canvas/public_html/lib/temp_cache.rb:30:in `enable'
/home/canvas/public_html/config/initializers/delayed_job.rb:152:in `block in <top (required)>'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/lifecycle.rb:88:in `block (2 levels) in add'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/lifecycle.rb:70:in `block in initialize'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/lifecycle.rb:88:in `block in add'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/lifecycle.rb:88:in `block in add'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/lifecycle.rb:88:in `block in add'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/lifecycle.rb:75:in `execute'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/lifecycle.rb:48:in `run_callbacks'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/worker.rb:223:in `perform'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/worker.rb:195:in `block (2 levels) in run'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/worker.rb:298:in `block in configure_for_job'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/worker.rb:53:in `running_job'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/worker.rb:294:in `configure_for_job'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/worker.rb:194:in `block in run'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/lifecycle.rb:70:in `block in initialize'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/lifecycle.rb:75:in `execute'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/lifecycle.rb:48:in `run_callbacks'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/worker.rb:187:in `run'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/switchman-inst-jobs-3.1.2/lib/switchman_inst_jobs/delayed/worker.rb:26:in `block in run'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/switchman-2.2.1/app/models/switchman/shard.rb:596:in `block in activate'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/switchman-2.2.1/app/models/switchman/shard.rb:98:in `activate'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/switchman-2.2.1/app/models/switchman/shard.rb:595:in `activate'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/switchman-inst-jobs-3.1.2/lib/switchman_inst_jobs/delayed/worker.rb:26:in `run'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/worker.rb:151:in `block in start'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/lifecycle.rb:70:in `block in initialize'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/lifecycle.rb:75:in `execute'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/lifecycle.rb:48:in `run_callbacks'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/worker.rb:149:in `start'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/switchman-inst-jobs-3.1.2/lib/switchman_inst_jobs/delayed/worker.rb:19:in `block in start'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/switchman-2.2.1/app/models/switchman/shard.rb:596:in `block in activate'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/switchman-2.2.1/app/models/switchman/shard.rb:98:in `activate'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/switchman-2.2.1/app/models/switchman/shard.rb:595:in `activate'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/switchman-inst-jobs-3.1.2/lib/switchman_inst_jobs/delayed/worker.rb:19:in `start'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/pool.rb:100:in `block in spawn_worker'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/pool.rb:112:in `block in fork_with_reconnects'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/pool.rb:108:in `fork'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/pool.rb:108:in `fork_with_reconnects'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/pool.rb:99:in `spawn_worker'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/pool.rb:77:in `block (2 levels) in spawn_all_workers'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/pool.rb:77:in `times'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/pool.rb:77:in `block in spawn_all_workers'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/pool.rb:76:in `each'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/pool.rb:76:in `spawn_all_workers'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/pool.rb:43:in `start'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/cli.rb:98:in `start'
/home/canvas/public_html/vendor/bundle/ruby/2.6.0/gems/inst-jobs-2.3.3/lib/delayed/cli.rb:50:in `run'
/home/canvas/public_html/script/delayed_job:11:in `<main>'

table messages_2021_42 exists but messages_2021_41 does not. public | messages | table | admin public | messages_2021_10 | table | postgres public | messages_2021_11 | table | postgres public | messages_2021_12 | table | postgres public | messages_2021_13 | table | postgres public | messages_2021_14 | table | postgres public | messages_2021_15 | table | postgres public | messages_2021_16 | table | postgres public | messages_2021_17 | table | postgres public | messages_2021_18 | table | postgres public | messages_2021_19 | table | postgres public | messages_2021_20 | table | postgres public | messages_2021_21 | table | postgres public | messages_2021_22 | table | postgres public | messages_2021_23 | table | postgres public | messages_2021_24 | table | postgres public | messages_2021_25 | table | postgres public | messages_2021_26 | table | postgres public | messages_2021_27 | table | postgres public | messages_2021_28 | table | postgres public | messages_2021_29 | table | postgres public | messages_2021_30 | table | postgres public | messages_2021_31 | table | postgres public | messages_2021_32 | table | postgres public | messages_2021_33 | table | postgres public | messages_2021_34 | table | postgres public | messages_2021_35 | table | postgres public | messages_2021_36 | table | postgres public | messages_2021_37 | table | postgres public | messages_2021_38 | table | postgres public | messages_2021_39 | table | canvas public | messages_2021_40 | table | canvas public | messages_2021_42 | table | canvas public | messages_2021_43 | table | canvas public | messages_2021_44 | table | canvas public | microsoft_sync_groups | table | canvas

roor0 commented 2 years ago

this is a partitioned table and generally gets created by a delayed_job, you can open up a rails console and run Messages::Partitioner.process to create them now.

amg-web commented 2 years ago

thanks for fix.

Only remain with the jobs which are in turn but do not processed. SQL Delayed::Backend::ActiveRecord::Job Load (2.0ms) WITH limited_jobs AS (SELECT id, ROW_NUMBER() OVER () AS row_number FROM (SELECT "delayed_jobs".* FROM "public"."delayed_jobs" WHERE (run_at<='2021-11-23 18:23:19.002533' AND locked_at IS NULL AND next_in_strand=TRUE) AND "delayed_jobs"."priority" BETWEEN 0 AND 10 AND "delayed_jobs"."queue" = 'canvas_queue' ORDER BY "delayed_jobs"."priority" ASC, "delayed_jobs"."run_at" ASC, "delayed_jobs"."id" ASC LIMIT 1 FOR UPDATE SKIP LOCKED) subquery) UPDATE "public"."delayed_jobs" SET locked_by = CASE row_number WHEN 1 THEN 'canvas.d.com:8943' END, locked_at = '2021-11-23 18:23:19.003552' FROM limited_jobs WHERE limited_jobs.id="public"."delayed_jobs".id RETURNING "public"."delayed_jobs".* [production:1 primary]

HOW TO FIX THIS? This job is not processinf even after fixing Messages::Partitioner.process

Ardena commented 2 years ago

This is a query to get the jobs that need to be processed next. Several are locked at once when one job worker is ready to pull new jobs, and then the next jobs should be processed in turn when another worker is ready to take up a job. Do you have an example of a job that isn't being processed?

amg-web commented 2 years ago

due 3 missing tables like messages_202141 jobs stay in Current Job list, they do not show any errors but never processed. After I use Messages::Partitioner.process new jobs started to work, but old are still in the Current list and stay there and number of not running jobs grows. What shell I do? ![delayed notification processing failed PG_UndefinedTable ERROR_ relation does not exist · Issue #1954 · instructure_canvas-lms - Google Chrome 25 11 2021 8_13_43](https://user-images.githubusercontent.com/22151881/143389501-96dc89e7-955d-4ab9-bb32-6c61f53c2b0c.png)

amg-web commented 2 years ago

this is a partitioned table and generally gets created by a delayed_job, you can open up a rails console and run Messages::Partitioner.process to create them now.

I have to run it again. why it was not created automatically?

irb(main):002:0> Messages::Partitioner.process
  SQL  (0.4ms)  BEGIN  [production:1 deploy]
  SQL  (0.5ms)  SET LOCAL statement_timeout=30000  [production:1 deploy]
  SQL  (29.7ms)   CREATE TABLE "public"."messages_2021_50" (
 LIKE "public"."messages" INCLUDING ALL,
 CHECK (created_at >= TIMESTAMP '2021-12-13 00:00:00'
AND
created_at < TIMESTAMP '2021-12-20 00:00:00'
)
 ) INHERITS ("public"."messages")
  [production:1 deploy]
  SQL  (1.2ms)  COMMIT  [production:1 deploy]
  SQL  (0.5ms)  BEGIN  [production:1 deploy]
  SQL  (0.4ms)  SET LOCAL statement_timeout=30000  [production:1 deploy]
  SQL  (11.8ms)   CREATE TABLE "public"."messages_2021_51" (
 LIKE "public"."messages" INCLUDING ALL,
 CHECK (created_at >= TIMESTAMP '2021-12-20 00:00:00'
AND
created_at < TIMESTAMP '2021-12-27 00:00:00'
)
 ) INHERITS ("public"."messages")
  [production:1 deploy]
  SQL  (1.0ms)  COMMIT  [production:1 deploy]
  SQL  (0.4ms)  BEGIN  [production:1 deploy]
  SQL  (0.4ms)  SET LOCAL statement_timeout=30000  [production:1 deploy]
  SQL  (12.2ms)   CREATE TABLE "public"."messages_2021_52" (
 LIKE "public"."messages" INCLUDING ALL,
 CHECK (created_at >= TIMESTAMP '2021-12-27 00:00:00'
AND
created_at < TIMESTAMP '2022-01-03 00:00:00'
)
 ) INHERITS ("public"."messages")
  [production:1 deploy]
  SQL  (1.1ms)  COMMIT  [production:1 deploy]
=> []
amg-web commented 2 years ago

fixed with help of developer trigger was missing in DB