search

collectiveidea / delayed_job

Database based asynchronous priority queue system -- Extracted from Shopify
http://groups.google.com/group/delayed_job
MIT License
4.81k stars 955 forks source link

Workers crash when Delayed::DeserializationError: ActiveRecord::RecordNotFound is raised #1157

Open edwinthinks opened 2 years ago

edwinthinks commented 2 years ago

Hello there,

We noticed that our workers when encountered with a delayed job that specifies a deleted object in its handler cause the workers to crash and halt processing more jobs.

Delayed::DeserializationError: ActiveRecord::RecordNotFound, class: CalendarRule, primary key: 485423 (Couldn't find SomeModel with 'id'=485423)
from /Users/emak/.rbenv/versions/2.7.4/lib/ruby/gems/2.7.0/gems/delayed_job-4.1.9/lib/delayed/psych_ext.rb:48:in `rescue in visit_Psych_Nodes_Mapping'
Caused by ActiveRecord::RecordNotFound: Couldn't find SomeModel with 'id'=485423
from /Users/emak/.rbenv/versions/2.7.4/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/relation/finder_methods.rb:357:in `raise_record_not_found_exception!'

It appears that this raise error is intentional but is I think it is odd that the worker would flat out crash and not just move on if there was an issue. Is there a way to make these errors occur but not crash our workers (we are using heroku dynos)

Any help would be great!

smudge commented 2 years ago

When executing jobs, the worker should rescue these errors here, but it's possible for deserialization failures to crash workers if a lifecycle hook (typically from a plugin) that runs outside of that run method attempts to deserialize a job (e.g. by calling payload_object). Is there a more complete stacktrace that might point to a particular plugin you are using?