Hello, we've been using delayed-job for years (mostly to send emails) with millions of jobs without a problem.
We recently updated to stay on LTS for Rails, and it might be a coincidence but we're now seeing small but intermittently reproducible errors where a scheduled job just decides to run early. I know, it makes no sense !
example:
Email_message scheduled to process_at Apr 18,2024 at 10:00 AM EDT, the job ran_at Feb 19, 2024 At 09:00 AM EST
Email_message scheduled to process_at Mar 01,2024 at 07:30 AM EST, the job ran_at Feb 19, 2024 At 07:00 AM EST
Mainly i'm trying to see if anyone has run into 'eager' job execution, and if they could point to anything that might cause it. I can't find a pattern of timezone, day of month, timestamp ( partial hours) ...but I do see batches of jobs queue up randomly on the same day (different users, whose jobs have both different created_at and updated_at times).
When it occurs I've confirmed that nobody has manually queued up the jobs through the UI by accident, and there was no restart of the server/process/workers on the same day (I had thought maybe something in our production CI deploy might be to blame). Maybe its the Leap year? :)
Update Details In the last update we went from Rails 6.0.4 to 6.1.7.6, we also updated delayed_job from 4.1.9 to 4.1.11
Current major packages
Rail - rails (~> 6.1.7.6)
Active Record - activerecord (6.1.7.6)
delayed_job (4.1.11)
delayed_job_active_record (4.1.7)
delayed_job_web (1.4.4)
I'm logging all future scheduled jobs for now, so we can see if the run_at times were mismtached with the email Process_at value (should be the same). Nothing so far, they all look in sync.
We could look at logging updates to the run_at on the jobs themselves next, to see if we can catch anything. I'm at a loss to know where to dig.
Hello, we've been using delayed-job for years (mostly to send emails) with millions of jobs without a problem.
We recently updated to stay on LTS for Rails, and it might be a coincidence but we're now seeing small but intermittently reproducible errors where a scheduled job just decides to run early. I know, it makes no sense !
example: Email_message scheduled to process_at Apr 18,2024 at 10:00 AM EDT, the job ran_at Feb 19, 2024 At 09:00 AM EST Email_message scheduled to process_at Mar 01,2024 at 07:30 AM EST, the job ran_at Feb 19, 2024 At 07:00 AM EST
Mainly i'm trying to see if anyone has run into 'eager' job execution, and if they could point to anything that might cause it. I can't find a pattern of timezone, day of month, timestamp ( partial hours) ...but I do see batches of jobs queue up randomly on the same day (different users, whose jobs have both different created_at and updated_at times).
When it occurs I've confirmed that nobody has manually queued up the jobs through the UI by accident, and there was no restart of the server/process/workers on the same day (I had thought maybe something in our production CI deploy might be to blame). Maybe its the Leap year? :)
Update Details In the last update we went from Rails 6.0.4 to 6.1.7.6, we also updated delayed_job from 4.1.9 to 4.1.11
Current major packages Rail - rails (~> 6.1.7.6) Active Record - activerecord (6.1.7.6) delayed_job (4.1.11) delayed_job_active_record (4.1.7) delayed_job_web (1.4.4)
I'm logging all future scheduled jobs for now, so we can see if the run_at times were mismtached with the email Process_at value (should be the same). Nothing so far, they all look in sync.
We could look at logging updates to the run_at on the jobs themselves next, to see if we can catch anything. I'm at a loss to know where to dig.