Que Scheduler not running

[piotrwodz]

Description

I defined two new jobs in que_schedule.yml. I deployed to staging environment and noticed Que::Scheduler::SchedulerJob is not picking up these new jobs because last time it was run was Fri, 24 Mar 2023 07:42:00.000000000 UTC +00:00.

I had a similar issue locally and what I did was:

1. I removed a que job with job class: Que::Scheduler::SchedulerJob
2. I run: Que::Scheduler::Migrations.reenqueue_scheduler_if_missing

This worked out locally but it does not fix an issue in staging environment. Any clues?

Versions

Please provide the following versions where you see the issue:

ruby version: ruby 3.0.5p211 que version: que (2.2.0) que-scheduler version: que-scheduler (4.4.0) postgres version: 11.16

@hlascelles Let me know if you have an idea what might be the reason, please.

[piotrwodz]

I tried to remove the job:

QueJob Destroy (3.4ms)  DELETE FROM "public"."que_jobs" WHERE "public"."que_jobs"."id" = $1  [["id", 868563]]
/var/app/current/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:672:in `exec_params': PG::RaiseException: ERROR:  Deletion of que_scheduler job prevented. Deleting the que_scheduler job is almost certainly a mistake. (ActiveRecord::StatementInvalid)
CONTEXT:  PL/pgSQL function que_scheduler_prevent_job_deletion() line 6 at RAISE
/var/app/current/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:672:in `exec_params': ERROR:  Deletion of que_scheduler job prevented. Deleting the que_scheduler job is almost certainly a mistake. (PG::RaiseException)
CONTEXT:  PL/pgSQL function que_scheduler_prevent_job_deletion() line 6 at RAISE```

[piotrwodz]

I found out what was causing it.

Que::Scheduler::SchedulerJob would not run due to job_schema_version: 1. I updated it to 2 and it started working again.

[piotrwodz]

I would expect the migration 7 to handle it. Nonetheless, it might help someone out down the road!

[hlascelles]

Ah interesting... Thanks for the writeup!

I've not used v2 in anger yet, only in development. When we do the upgrade I'll see if I can replicate the issue. But this issue will be good documentation until then.

Thanks again.

[Faq]

I had the same problem with job_schema_version: 1, changed to 2 and all worked.. que (2.2.0), que-scheduler (4.4.0), ruby 3.2.2, rails 7.0.4.3 Was expecting that this is handled, but seems not. Did not found where this is documented.

[kgreunke]

I'm not sure if this should be fixed but it is not. Migrating to 8 does not update the existing scheduler job schema version, and because of the database constraints you can't delete the job and re-enqueue it easily.

[hlascelles]

Looking at que's docs on this, it doesn't seem to talk about the need to update this value when upgrading, but it appears it is needed.

We needed to run this for all of our jobs to work:

        UPDATE que_jobs
        SET job_schema_version = 2
        WHERE job_schema_version IS DISTINCT FROM 2

This is nothing to do with que-scheduler in particular.

I'll raise an issue with que to see if there needs to be more visibility on this.

[kgreunke]

Ya, sorry for the ping. I understand that this is very likely a Que issue now. I just landed here because the que-scheduler job was the only thing not working. Chance I guess. Thank you @hlascelles and @piotrwodz for the way forward. I was able to get my jobs fixed up and working again.