Deserialization errors after upgrading from Rails 4.2 to 5.1+

[drewB]

I am working on upgrading an app from Rails 4.2 to 5.2. I have am running into an issue were jobs that were created in 4.2 are raising errors when they are invoked under Rails 5.2.

Delayed::DeserializationError (Job failed to load: not delegated...

I have narrowed it down to a problem after moving from 5.0 to 5.1. In 5.0.7 there is no problem but there is in 5.1.0. I can reproduce on a simple test case (taken from job.handler) by doing YAML.load(yml) where yml:

object: !ruby/object:Account
  raw_attributes:
    id: '8469'
  attributes: !ruby/object:ActiveRecord::AttributeSet
    attributes: !ruby/object:ActiveRecord::LazyAttributeHash
      types:
        id: &4 !ruby/object:ActiveRecord::Type::Integer
          precision: 
          scale: 
          limit: 8
          range: !ruby/range
            begin: -9223372036854775808
            end: 9223372036854775808
            excl: true
      values:
        id: '8469'
        created_at: '2019-11-15 21:16:15.257401'
      additional_types: {}
      materialized: true
      delegate_hash:
        id: !ruby/object:ActiveRecord::Attribute::FromDatabase
          name: id
          value_before_type_cast: '8469'
          type: *4
          value: 8469
        created_at: !ruby/object:ActiveRecord::Attribute::FromDatabase
          name: created_at
          value_before_type_cast: '2019-11-15 21:16:15.257401'
          type: !ruby/object:ActiveRecord::AttributeMethods::TimeZoneConversion::TimeZoneConverter
            subtype: !ruby/object:ActiveRecord::ConnectionAdapters::PostgreSQL::OID::DateTime
              precision: 
              scale: 
              limit: 
          value: 2019-11-15 21:16:15.257401000 Z
  new_record: false
  active_record_yaml_version: 0

That gives the error ArgumentError (not delegated). I have found that removing the subtype under created_at makes the problem go away but not idea why. I have tried changing the subtype to something simple like an integer and get the same problem.

Anyone have thoughts on how to approach this? I am really surprised that I have not found any info on others running into the same problem.

[albus522]

The problem boils down to newer Rails can't deserialize the ActiveRecord YAML produced by older versions. I used this augmentation to the Ruby YAML parser to be able to load and reserialize the old yaml objects https://gist.github.com/albus522/ba5bb04205a9dc316304f8f228adda2e It worked for what I needed at the time but I am quite sure it is not a complete solution.

[drewB]

@albus522 thanks for you response and shared code. Seem odd that new Rails can't deserialize since there is a active_record_yaml_version attribute. The current plan I am exploring is to dual boot rails 5.1 and 5.2 using https://github.com/Shopify/bootboot and then reserialize all the handlers in 5.1 mode. Something like:

Delayed::Job.all.find_each do |job|
   data = YAML.load_dj(job.handler)
   job.update(handler: YAML.dump(data))
end

[albus522]

That gem doesn't load 2 versions in the same process. It is the same as dual booting a computer. You don't run 2 versions at the same time, you make it easier to switch between the two.

So all your code snippet will do is reserialize the same old data.

[drewB]

It doesn't need two versions in the same process. On 4.2, all jobs are serialized with active_record_yaml_version : 0. On Rails 5.1, it seems to be able to correctly deserialize the job (I have only tested a few so far) and when you reserialize it uses the active_record_yaml_version : 1 format. The dual boot is just so the rake task I create for this can run in 5.1 while the rest of the app runs in 5.2.

[drewB]

My idea didn't end up working. While the test job could be deserialized, other ones ran into problems related to undefined things in Rails 5.1. @albus522, if I understand the gist you linked correctly, you are basically ignoring the ruby/object:ActiveRecord::AttributeSet in the handler which is where all the extra method and data types are defined and instead just grabbing the id from raw_attributes and finding the obj. Is that correct?

You said, "I am quite sure it is not a complete solution." Were there specific areas you were concerned about or just that you were focused on a narrow set so aren't sure if there might be other issues?

[albus522]

Yes and yes. The augmented parser essentially halts the parsing early in the process so that it doesn't try to deserialize the data for classes that don't exist. Deserializing the ActiveRecord object by doing a database lookup is already what DJ does, but DJ doesn't halt the parsing process, it hooks into the tail end of the deserialization pulling a clean copy of the database object. While the approach is similar enough to what DJ does by default, the data I used this on was simple enough to not present much of a challenge.

[drewB]

@albus522 thanks again for your help. I created a migration to handle the upgrade and have tested it on our full data set and everything looks good. Only adjustment to the gist you shared I made was to handle when raw_attributes doesn't exist. We had some really old failed jobs from Rails 3 that didn't have that attribute.

Here is the migration for those that might be interested https://gist.github.com/drewB/2bcb5448828b2ce95021f63b4dccd5d5