jdmcd
commented
3 years ago Hi there! Thanks for raising this issue. You are right, the queue executing on multiple event loops is intentional/by design. There is a way around it, let me know if this makes sense:
- You can create a new queue type,
completion, that your completion jobs are assigned to (i.e.app.queues(.completion).dispatch(...) - You can create a new worker with
./Run queues --queue completionthat will only run jobs assigned to thecompletionqueue - Give that worker only 1 event loop - either through restricting the number of CPUs on the machine to 1 or setting up some kind of environment variable system where you adjust the number of eventloops in
main.swiftbased on some arbitrary env var value
0xTim
leonidas-o
**Is your feature request related to a problem? Currently, when running a Vapor Queue on a Mac in development, when adding jobs to a queue the jobs are pulled off by multiple eventloops, (assumption) and thus the jobs don't complete in added order. Thus the queue is not FIFO.
Describe the solution you'd like I would like a setting to make a queue sequential or FIFO.
Describe alternatives you've considered I can't find any alternatives. Thus I'm blocked with how I can use a Vapor Queue in this critical instance.
Additional context I understand the value of a queue being processed by multiple event loops. I have other use cases that use multi event-loops successfully. But the need for FIFO is critical, in a different use case.
Specifically what I need: I have a series of jobs that save large amounts of data to a postgres database using Fluent. It is fine that multiple event loops process those jobs because all the save are independent. But here is the kicker. I need to add a completion job to that queue, that executes after all the other jobs complete. As it now stands, that final job, added to the queues last, executes before some of the dependent jobs complete.
A sequential/FIFO is an easy way to ensure the final job is executed after all other jobs complete.