rclimepoint commented 1 year ago

Description

The Que scheduler migration calls Que::Scheduler::VersionSupport.enqueue_a_job(Que::Scheduler::SchedulerJob) when applying the first version of Que scheduler. This is not a problem if you run this migration after you have migrated Que, or if your database schema is already up-to-date and doesn't need to run all the previous migrations.

However, when starting off with an empty database, the db:migrate will start migrating from the first version. Since this is an upgrade, the Que migration will be migrated last and this results in the Que scheduler migration failing due to the 'kwargs' column not existing in the que_jobs table at this point.

Versions

ruby version: 2.7.6 que version: 2.2 que-scheduler version: 4.4 postgres version: 14.5

Messages

An example below is if you have following migration files:

20211201000000_create_que_scheduler_schema.rb

class CreateQueSchedulerSchema < ActiveRecord::Migration[6.0]
 def change
   Que::Scheduler::Migrations.migrate!(version: 6)
 end
end

20221201000000_update_que7.rb

class UpdateQue7 < ActiveRecord::Migration[7.0]
 def change
   Que.migrate!(version: 7)
 end
end

Caused by: PG::UndefinedColumn: ERROR: column "kwargs" of relation "que_jobs" does not exist LINE 1: ...e_jobs (queue, priority, run_at, job_class, args, kwargs, da...

(eval):9:in `run_sql_middleware'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-2.2.0/lib/que/connection.rb:60:in `execute'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-2.2.0/lib/que/connection_pool.rb:57:in `block in execute'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-2.2.0/lib/que/connection_pool.rb:39:in `block in checkout'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-2.2.0/lib/que/active_record/connection.rb:13:in `block (2 levels) in checkout'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/activerecord-7.0.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:215:in `with_connection'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-2.2.0/lib/que/active_record/connection.rb:12:in `block in checkout'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/activesupport-7.0.4/lib/active_support/execution_wrapper.rb:88:in `wrap'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-2.2.0/lib/que/active_record/connection.rb:23:in `wrap_in_rails_executor'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-2.2.0/lib/que/active_record/connection.rb:11:in `checkout'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-2.2.0/lib/que/connection_pool.rb:18:in `checkout'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-2.2.0/lib/que/connection_pool.rb:57:in `execute'
/usr/local/rbenv/versions/2.7.6/lib/ruby/2.7.0/forwardable.rb:235:in `execute'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-2.2.0/lib/que/job.rb:131:in `enqueue'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-scheduler-4.4.0/lib/que/scheduler/version_support.rb:57:in `enqueue_a_job'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-scheduler-4.4.0/lib/que/scheduler/migrations.rb:55:in `migrate_up'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-scheduler-4.4.0/lib/que/scheduler/migrations.rb:22:in `block in migrate!'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-scheduler-4.4.0/lib/que/scheduler/db.rb:23:in `block in transaction'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-2.2.0/lib/que/utils/transactions.rb:14:in `block in transaction'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-2.2.0/lib/que/connection_pool.rb:39:in `block in checkout'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-2.2.0/lib/que/active_record/connection.rb:13:in `block (2 levels) in checkout'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/activerecord-7.0.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:215:in `with_connection'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-2.2.0/lib/que/active_record/connection.rb:12:in `block in checkout'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/activesupport-7.0.4/lib/active_support/execution_wrapper.rb:92:in `wrap'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-2.2.0/lib/que/active_record/connection.rb:23:in `wrap_in_rails_executor'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-2.2.0/lib/que/active_record/connection.rb:11:in `checkout'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-2.2.0/lib/que/connection_pool.rb:18:in `checkout'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-2.2.0/lib/que/utils/transactions.rb:12:in `transaction'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-scheduler-4.4.0/lib/que/scheduler/db.rb:23:in `call'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-scheduler-4.4.0/lib/que/scheduler/db.rb:23:in `transaction'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-scheduler-4.4.0/lib/que/scheduler/migrations.rb:19:in `migrate!'
/opt/xxx/db/migrate/20211201000000_create_que_scheduler_schema.rb:3:in `change'

Workaround

We've added a monkey patch to skip the job enqueue when doing a migration, and then re-enqueue the scheduler once we've migrated Que to the latest version:

module Que
  module Scheduler
    module Migrations
      class << self
        private

        def migrate_up(current, version)
          execute_step((current += 1), :up) until current == version
        end
      end
    end
  end
end

20221201000000_update_que7.rb

class UpdateQue7 < ActiveRecord::Migration[7.0]
 def change
   Que.migrate!(version: 7)
   Que::Scheduler::Migrations.reenqueue_scheduler_if_missing
 end
end

hlascelles commented 1 year ago

Thank you for the excellent report @rclimepoint!

Yes, I can see how that would be an issue. Hmm, this is a tough one - how to know if there are "more migrations to come". I guess maybe for this exact error I can make it give a precise error message with instructions of what to do.

Will have a look...

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

hlascelles commented 1 year ago

Will have a look at this one still. I think it isn't urgent, but a better error message would be nice.

blunckr-aj commented 1 year ago

It seems like the only reliable way to solve this might be to not attempt to enqueue the job during migrations at all. Instead you could add a step to the instructions to enqueue it after migrations, either in seeds or a rake task.

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

hlascelles commented 1 year ago

I will look at this...

stale[bot] commented 5 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

hlascelles / que-scheduler

Upgrading to Que 2.x from 1.4 will cause the Que scheduler migration to fail on prior migrations #381

Description

Versions

Messages

Workaround