Delayed jobs are not being picked up by the workers

instructure / canvas-lms

The open LMS by Instructure, Inc.

https://github.com/instructure/canvas-lms/wiki

GNU Affero General Public License v3.0

5.58k stars 2.48k forks source link

Delayed jobs are not being picked up by the workers #2155

Open maksimf opened 1 year ago

maksimf commented 1 year ago

Summary:

I have a few hundred tasks, just sitting there, not being picked up by any of the workers. I did check that:

tasks are in the same queue that the workers are launched for (canvas_queue)
workers are indeed spawned (via ps aux)

photo_2023-02-07 22 04 55

Steps to reproduce:

Perform any action which triggers a delayed job, like inviting new user

Expected behavior:

Email is sent

Actual behavior:

Email job is queued, but the task is not picked up by the worker (0/1 attempts)

maksimf commented 1 year ago

I have updated all the stuck jobs with locked_by = 'RANDOM STRING' and locked_at = NOW(), after that I have removed all those jobs and this, somehow, made jobs being picked up again. But I would still want to know why this might have happened

KarimElagamy commented 9 months ago

Hi, I'm running into this same exact issue right now, any chance you could post a more detailed explanation of how you fixed this? Would be much appreciated