"no _dump_data is defined for class Proc" errors intermittently generated on page loads

svgit1 commented 3 years ago

Summary:

We're trying to determine what's causing intermittent 500 response errors resulting in users seeing the error page ("Page Error. Something broke unexpectedly."). Note that this is intermittent where a simple page refresh resolves. We can't easily replicate but this is frequent enough that these errors being generated multiple times per day for users.

Our frontends are running canvas-lms\stable from 2020-07-15 (f795ec2464)

Steps to reproduce:

Navigate to any page in Canvas. Page intermittently fails to load (500 response) and simple refresh resolves.

Expected behavior:

Page loads without issues every time.

Actual behavior:

Page intermittently fails to load, users see the error page ("Page Error. Something broke unexpectedly."), and "no _dump_data is defined for class Proc" error report is generated (see below details).

Additional notes:

Error report that's generated:

Message: no _dump_data is defined for class Proc

Category: TypeError Backtrace:

/var/canvas/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.3/lib/active_support/cache.rb:784:in `dump'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.3/lib/active_support/cache.rb:784:in `compress!'
/var/canvas/config/initializers/active_support.rb:54:in `compress!'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.3/lib/active_support/cache.rb:722:in `initialize'
/var/canvas/lib/canvas/cache_register.rb:219:in `new'
/var/canvas/lib/canvas/cache_register.rb:219:in `block in fetch_with_cache_register'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.3/lib/active_support/cache.rb:663:in `block in instrument'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.3/lib/active_support/notifications.rb:170:in `instrument'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.3/lib/active_support/cache.rb:663:in `instrument'
/var/canvas/lib/canvas/cache_register.rb:218:in `fetch_with_cache_register'
/var/canvas/lib/canvas/cache_register.rb:169:in `fetch_with_batched_keys'
/var/canvas/lib/assignment_override_applicator.rb:88:in `block in overrides_for_assignment_and_user'
/var/canvas/lib/temp_cache.rb:50:in `cache'
/var/canvas/lib/assignment_override_applicator.rb:87:in `overrides_for_assignment_and_user'
/var/canvas/lib/assignment_override_applicator.rb:36:in `assignment_overridden_for'
/var/canvas/lib/dates_overridable.rb:42:in `overridden_for'
/var/canvas/lib/dates_overridable.rb:255:in `block in context_module_tag_info'
/var/canvas/lib/canvas/cache_register.rb:217:in `block in fetch_with_cache_register'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.3/lib/active_support/cache.rb:663:in `block in instrument'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.3/lib/active_support/notifications.rb:168:in `block in instrument'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.3/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.3/lib/active_support/notifications.rb:168:in `instrument'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.3/lib/active_support/cache.rb:663:in `instrument'
/var/canvas/lib/canvas/cache_register.rb:217:in `fetch_with_cache_register'
/var/canvas/lib/canvas/cache_register.rb:169:in `fetch_with_batched_keys'
/var/canvas/lib/dates_overridable.rb:245:in `context_module_tag_info'
/var/canvas/app/controllers/context_modules_controller.rb:349:in `block in content_tag_assignment_data'
/var/canvas/app/controllers/context_modules_controller.rb:347:in `each'
/var/canvas/app/controllers/context_modules_controller.rb:347:in `content_tag_assignment_data'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.4.3/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.4.3/lib/abstract_controller/base.rb:194:in `process_action'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.4.3/lib/action_controller/metal/rendering.rb:30:in `process_action'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.4.3/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.3/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
/var/canvas/app/controllers/application_controller.rb:561:in `report_to_datadog'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.3/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/inst_statsd-2.1.6/lib/inst_statsd/statsd.rb:95:in `batch'
/var/canvas/app/controllers/application_controller.rb:549:in `batch_statsd'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.3/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/var/canvas/lib/temp_cache.rb:28:in `enable'
/var/canvas/app/controllers/application_controller.rb:545:in `enable_request_cache'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.3/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/var/canvas/app/controllers/application_controller.rb:539:in `set_locale'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.3/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.3/lib/active_support/callbacks.rb:136:in `run_callbacks'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.4.3/lib/abstract_controller/callbacks.rb:41:in `process_action'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.4.3/lib/action_controller/metal/rescue.rb:22:in `process_action'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.4.3/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.3/lib/active_support/notifications.rb:168:in `block in instrument'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.3/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.3/lib/active_support/notifications.rb:168:in `instrument'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.4.3/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.4.3/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.4.3/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.4.3/lib/abstract_controller/base.rb:134:in `process'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/actionview-5.2.4.3/lib/action_view/rendering.rb:32:in `process'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.4.3/lib/action_controller/metal.rb:191:in `dispatch'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.4.3/lib/action_controller/metal.rb:252:in `dispatch'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.4.3/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.4.3/lib/action_dispatch/routing/route_set.rb:34:in `serve'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.4.3/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.4.3/lib/action_dispatch/journey/router.rb:35:in `each'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.4.3/lib/action_dispatch/journey/router.rb:35:in `serve'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.4.3/lib/action_dispatch/routing/route_set.rb:840:in `call'
/var/canvas/gems/plugins/respondus_soap_endpoint/lib/respondus_soap_endpoint/middleware.rb:78:in `call'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/rack-2.2.3/lib/rack/deflater.rb:44:in `call'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/rack-2.2.3/lib/rack/chunked.rb:98:in `call'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/rack-2.2.3/lib/rack/tempfile_reaper.rb:15:in `call'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/rack-2.2.3/lib/rack/etag.rb:27:in `call'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/rack-2.2.3/lib/rack/conditional_get.rb:27:in `call'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
/var/canvas/app/middleware/request_throttle.rb:63:in `block in call'
/var/canvas/app/middleware/request_throttle.rb:314:in `reserve_capacity'
/var/canvas/app/middleware/request_throttle.rb:58:in `call'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.4.3/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
/var/canvas/app/middleware/request_context_session.rb:25:in `call'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:266:in `context'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:260:in `call'
/var/canvas/app/middleware/load_account.rb:29:in `call'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.4.3/lib/action_dispatch/middleware/cookies.rb:670:in `call'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.4.3/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.3/lib/active_support/callbacks.rb:98:in `run_callbacks'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.4.3/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.4.3/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.4.3/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/railties-5.2.4.3/lib/rails/rack/logger.rb:38:in `call_app'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/railties-5.2.4.3/lib/rails/rack/logger.rb:28:in `call'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.4.3/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/request_store-1.5.0/lib/request_store/middleware.rb:19:in `call'
/var/canvas/app/middleware/request_context_generator.rb:49:in `call'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
/var/canvas/app/middleware/prevent_non_multipart_parse.rb:33:in `call'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/rack-2.2.3/lib/rack/runtime.rb:22:in `call'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.3/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.4.3/lib/action_dispatch/middleware/executor.rb:14:in `call'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/rack-2.2.3/lib/rack/sendfile.rb:110:in `call'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/railties-5.2.4.3/lib/rails/engine.rb:524:in `call'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/railties-5.2.4.3/lib/rails/railtie.rb:190:in `public_send'
/var/canvas/vendor/bundle/ruby/2.6.0/gems/railties-5.2.4.3/lib/rails/railtie.rb:190:in `method_missing'
/usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:107:in `process_request'
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request'
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop'
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads'
/usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception'

svgit1 commented 3 years ago

Bumping this - does anyone have any thoughts on where we should investigate?

Versions running: apache: 2.4.29 2020-03-13T12:26:16 bundler: 2.1.4 rails (part of bundler): 5.2.4.3 node: 10.22.0 python: 2.7.17 ruby: ruby 2.6.6p146 (2020-03-31 revision 67876) yarn: 1.10.1. Held at this version per the yarn section in Production Start guide. I've also tried the latest version of yarn, didn't help this error. passenger: 6.0.6. Tried an older version of passenger, didn't help this error.

Redis cache: 6.0.5

dluhhbiu commented 3 years ago

I have the same error when visiting the settings page of courses. It`s a random error and I can't replicate it with 100%. Just visit the settings page several times.

svgit1 commented 3 years ago

I have the same error when visiting the settings page of courses. It`s a random error and I can't replicate it with 100%. Just visit the settings page several times.

Same here. We've noticed the Settings page in courses replicates this error the most. We've seen it elsewhere too like during quizzes, assignments, etc., but the settings page seems to be where it occurs the most. Note that replicating has been very hard for us, the percent is quite low but enough to be a problem - we can't replicate consistently.

jfoscue commented 1 year ago

Bumping this issue. We are lots of these in our self deployed canvas

rezacademic commented 13 hours ago

Bumping this again, we see this happening frequently on settings pages

instructure / canvas-lms