search

rq / rq-scheduler

A lightweight library that adds job scheduling capabilities to RQ (Redis Queue)
MIT License
1.45k stars 230 forks source link

What would cause a job to get removed from the scheduler? #244

Closed taewookim closed 4 years ago

taewookim commented 4 years ago

Since my project has so many moving parts.. probably best to explain the symptom

I have 1 scheduler running on 1 queue. I add scheduled jobs ( to be executed within seconds, i.e.. they have all similar start times).

I keep repeating scheduling of jobs with NO rq worker doing anything (in fact, the process is completely off). In another words, the queue should just be piling up.

But ALL of a sudden.. the queue gets chopped off (randomly) and first 70-80% of jobs just disappear.

Does this have anything to do with: 1) the "max length" of queue? (but i dont recall seeing any limits) 2) does the scheduler automatically "discard" jobs where the start time is BEFORE the current time?

taewookim commented 4 years ago

@selwin

If #2 is the case...

Suppose i have a scenario where 2 jobs are scheduled at exact same time and take a "while"... is it safet to assume that i need minimum of 2 RQ workers?

otherwise, the one that's scheduled but doesn't get picked up by the worker will eventually get removed b/c the start date has been passed?

And is there an efficient way of having RQ scheduler somehow "space out" jobs so that they don't schedule jobs at exact same time?

taewookim commented 4 years ago

ran my own experiment. RQ scheduler does indeed remove jobs whose start date < now.