Active Record Migration Failure

[RaviKumarPesala]

Hi @ankane ,

We have implemented lockbox encryption for around 20 models with almost 200 attributes.

Now we're trying to migrate the existing data to encrypted, and we have more than 10 million records in production. So if we try to write any migration script to migrate all that data, Are there any chances of migration fail due to huge number of records.

• Just wanted to understand how the lockbox handles such migration failure scenarios ?
• How lockbox handles state management of attributes?
• Can we write rake tasks to migrate the existing data ?

Is there any other approach you'd suggest how to migrate such huge data. We have to be sure that lockbox will handle these scenarios.

Planned approach: We're planning to create an Application Job or an Iteration Job that can run the migration which can be paused/started anytime. Is this an approach we can proceed with? or do you see any drawbacks in it ?

Thanks, Ravi Kumar

[ankane]

Hey @RaviKumarPesala, Lockbox provides a migrate method, but you could also write your own or build on top of it if you're trying to parallelize it. You can interrupt the method to see how it handles failures and state management.

[RaviKumarPesala]

@ankane , Thanks for the response.

Could you please suggest any approach on how to override the migrate method to see the failures and state management.

[kalemi19]

@ankane interested in knowing about the override, too. Migrating 300K+ records could've been done faster with Sidekiq.

[kalemi19]

For anyone wondering, this is how I'm migrating 300K+ records using Sidekiq. I'm also skipping callbacks.

module Users
  class EncryptWorker
    include Sidekiq::Worker

    sidekiq_options queue: :low

    def perform(ids = nil)
      if ids.nil?
        ids = User.collection.aggregate([]).map { |oid| oid.as_json['_id']['$oid'] }

        ids.each_slice(50) do |batch|
          Users::EncryptWorker.perform_async(batch)
        end
      else
        fields = User.respond_to?(:lockbox_attributes) ? User.lockbox_attributes.select { |k, v| v[:migrating] } : {}
        blind_indexes = User.respond_to?(:blind_indexes) ? User.blind_indexes.select { |k, v| v[:migrating] } : {}

        users = User.where(id: { '$in': ids })
        users.update_all(encrypting: true) # Prevent callbacks from being fired

        users.each do |user|
          fields.each do |k, v|
            user.send("#{v[:attribute]}=", user.send(k)) unless user.send(v[:encrypted_attribute])
          end

          # with Blind Index 2.0, bidx_attribute should be already set for each record
          blind_indexes.each do |k, v|
            user.send("compute_#{k}_bidx") unless user.send(v[:bidx_attribute])
          end

          next unless user.changed?
          user.save!(validate: false)
        end

        users.update_all('$unset' => { encrypting: '' }) # Restoring callbacks
      end
    end
  end
end