Closed salmonsteak1 closed 1 week ago
🤔 You're enqueuing the jobs sequentially, no? And the ones enqueued first are these:
TestSqJobs::SemiUrgentPriorityJobA,
TestSqJobs::SemiUrgentPriorityJobB,
TestSqJobs::SemiUrgentPriorityJobC,
So they're going to be run before
TestSqJobs::UrgentPriorityJobA,
TestSqJobs::UrgentPriorityJobB,
TestSqJobs::UrgentPriorityJobC,
because when they're enqueued, they're the only ones there.
Ah, I see what you mean. Yes, there's a bug there in the order of queues when they use prefixes. The order is the one returned from the DB from this query:
relation.where(([ "queue_name LIKE ?" ] * prefixes.count).join(" OR "), *prefixes).distinct(:queue_name).pluck(:queue_name)
Which doesn't need to match the order in the LIKE
clause. These should be reordered to match the order in the clause and the config.
This should work fine if you don't use prefixes, though (which is what I'd recommend in this case, anyway, collapsing all your low_
, urgent_
, etc. queues into one). I'll fix the case with prefixes.
I see, thanks for the info @rosa! I was thinking to use wildcards together with the priority keywords so that I still can have separate queues for specific jobs while still having the benefit of using queue priorities.
Can I confirm what will be the scheduling algorithm used within jobs that match the same wildcard queue name? So, for example my queue.yml
file would have queues defined as: [urgent*, semi_urgent*, default*, low*]
, and I have urgent_A
, urgent_B
and urgent_C
jobs enqueued. Will the selection of these jobs be FCFS? Or would it be by random selection? Thanks!
Within the same wildcard name, urgent_A, urgent_B, urgent_C
, the selection would be non-deterministic, it'll depend on the order the DB returns the queue names from SELECT queue_name FROM solid_queue_ready_executions WHERE queue_name LIKE 'urgent_%'
).
If you use exact queue names, polling will be faster, and the order will be deterministic.
The fix for this was released in version 1.0.2, thanks again for the report!
Hey there, I'm testing out the queue priorities in solid queue. My
queue.yml
is as follows:I also have 3 test jobs for each queue priority level - A, B and C:
Finally, I execute each job 5 times using the following ruby code:
I've observed this on mission control, and it seems like not all the jobs with the urgent prefix gets fulfilled first. Here's a screenshot of my observations:
We can see that jobs are taking from the
default
andsemi_urgent
queue instead of clearing out the job queues with theurgent
prefix.