search

instructure / canvas-lms

The open LMS by Instructure, Inc.
https://github.com/instructure/canvas-lms/wiki
GNU Affero General Public License v3.0
5.61k stars 2.48k forks source link

MediaObject#updated_viewed_at_and_retrieve_details fails with error in canvas_kaltura with no Kaltura server configured #1747

Open grahamb opened 3 years ago

grahamb commented 3 years ago

Summary:

Since deploying the October 7 release, we have been experiencing repeated failed jobs related to the canvas_kaltura gen. Specifically, MediaObject#updated_viewed_at_and_retrieve_details jobs are failing with an error of undefined method [] for nil:NilClass in gems/canvas_kaltura/lib/canvas_kaltura/kaltura_client_v3.rb:41.

Steps to reproduce:

  1. Do not have a Kaltura server configured
  2. Be running the Oct 7 release or later
  3. There is no step 3.

Expected behavior:

Jobs should not fail.

Actual behavior:

MediaObject#updated_viewed_at_and_retrieve_details fail.

Additional notes:

Sample stack trace:

undefined method `[]' for nil:NilClass
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/gems/canvas_kaltura/lib/canvas_kaltura/kaltura_client_v3.rb:41:in `initialize'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/app/models/media_object.rb:236:in `new'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/app/models/media_object.rb:236:in `retrieve_details'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/app/models/media_object.rb:289:in `updated_viewed_at_and_retrieve_details'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/performable_method.rb:23:in `perform'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/backend/base.rb:254:in `block in invoke_job'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/lifecycle.rb:67:in `block in initialize'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/lifecycle.rb:72:in `execute'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/lifecycle.rb:45:in `run_callbacks'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/backend/base.rb:251:in `invoke_job'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/switchman-inst-jobs-2.0.0/lib/switchman_inst_jobs/delayed/backend/base.rb:85:in `block in invoke_job'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/switchman-2.0.0/app/models/switchman/shard.rb:628:in `block in activate'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/switchman-2.0.0/app/models/switchman/shard.rb:96:in `activate'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/switchman-2.0.0/app/models/switchman/shard.rb:627:in `activate'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/switchman-inst-jobs-2.0.0/lib/switchman_inst_jobs/delayed/backend/base.rb:85:in `invoke_job'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/worker.rb:210:in `block (2 levels) in perform'
/usr/local/lib/ruby/2.6.0/benchmark.rb:308:in `realtime'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/worker.rb:204:in `block in perform'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-statsd-1.4.1/lib/inst_jobs_statsd/stats/timing/perform.rb:22:in `block (2 levels) in enable_perform_timing'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-statsd-1.4.1/lib/inst_jobs_statsd/stats/timing.rb:8:in `block in report_timing'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst_statsd-2.1.6/lib/inst_statsd/statsd.rb:89:in `time'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-statsd-1.4.1/lib/inst_jobs_statsd/stats/timing.rb:8:in `report_timing'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-statsd-1.4.1/lib/inst_jobs_statsd/stats/timing/perform.rb:21:in `block in enable_perform_timing'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/lifecycle.rb:85:in `block (2 levels) in add'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-statsd-1.4.1/lib/inst_jobs_statsd/stats/timing/perform.rb:13:in `block (2 levels) in enable_batching'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst_statsd-2.1.6/lib/inst_statsd/statsd.rb:98:in `block in batch'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/statsd-ruby-1.4.0/lib/statsd.rb:73:in `easy'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/statsd-ruby-1.4.0/lib/statsd.rb:429:in `batch'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst_statsd-2.1.6/lib/inst_statsd/statsd.rb:96:in `batch'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-statsd-1.4.1/lib/inst_jobs_statsd/stats/timing/perform.rb:12:in `block in enable_batching'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/lifecycle.rb:85:in `block (2 levels) in add'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/config/initializers/delayed_job.rb:151:in `block (2 levels) in <top (required)>'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/lib/temp_cache.rb:28:in `enable'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/config/initializers/delayed_job.rb:150:in `block in <top (required)>'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/lifecycle.rb:85:in `block (2 levels) in add'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/lifecycle.rb:67:in `block in initialize'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/lifecycle.rb:85:in `block in add'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/lifecycle.rb:85:in `block in add'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/lifecycle.rb:85:in `block in add'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/lifecycle.rb:72:in `execute'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/lifecycle.rb:45:in `run_callbacks'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/worker.rb:201:in `perform'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/worker.rb:174:in `block (2 levels) in run'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/worker.rb:271:in `block in configure_for_job'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/worker.rb:40:in `running_job'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/worker.rb:267:in `configure_for_job'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/worker.rb:173:in `block in run'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/lifecycle.rb:67:in `block in initialize'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/lifecycle.rb:72:in `execute'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/lifecycle.rb:45:in `run_callbacks'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/worker.rb:166:in `run'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/switchman-inst-jobs-2.0.0/lib/switchman_inst_jobs/delayed/worker.rb:26:in `block in run'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/switchman-2.0.0/app/models/switchman/shard.rb:628:in `block in activate'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/switchman-2.0.0/app/models/switchman/shard.rb:96:in `activate'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/switchman-2.0.0/app/models/switchman/shard.rb:627:in `activate'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/switchman-inst-jobs-2.0.0/lib/switchman_inst_jobs/delayed/worker.rb:26:in `run'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/worker.rb:133:in `block in start'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/lifecycle.rb:67:in `block in initialize'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/lifecycle.rb:72:in `execute'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/lifecycle.rb:45:in `run_callbacks'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/worker.rb:131:in `start'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/switchman-inst-jobs-2.0.0/lib/switchman_inst_jobs/delayed/worker.rb:19:in `block in start'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/switchman-2.0.0/lib/switchman/default_shard.rb:7:in `activate'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/switchman-inst-jobs-2.0.0/lib/switchman_inst_jobs/delayed/worker.rb:19:in `start'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/pool.rb:97:in `block in spawn_worker'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/pool.rb:108:in `block in fork_with_reconnects'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/pool.rb:105:in `fork'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/pool.rb:105:in `fork_with_reconnects'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/pool.rb:96:in `spawn_worker'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/pool.rb:74:in `block (2 levels) in spawn_all_workers'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/pool.rb:74:in `times'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/pool.rb:74:in `block in spawn_all_workers'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/pool.rb:73:in `each'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/pool.rb:73:in `spawn_all_workers'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/pool.rb:40:in `start'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/cli.rb:96:in `start'
/var/rails/canvas/releases/20201105194423-sfu-release-2020-11-05-5/vendor/bundle/ruby/2.6.0/gems/inst-jobs-0.16.0/lib/delayed/cli.rb:34:in `run'
/var/rails/canvas/current/script/delayed_job:11:in `<main>'
xcudder commented 2 years ago

@grahamb Did you ever have the Kaltura plugin enabled, configured and later disabled? Do you happen to know what job triggered this?

grahamb commented 2 years ago

The error was being triggered by MediaObject#updated_viewed_at_and_retrieve_details, as mentioned in my post. Kaltura may have been enabled at some point; this issue is a couple years old now so I don't remember all the details.