search

NicolasLM / spinach

Modern Redis task queue for Python 3
https://spinach.readthedocs.io
BSD 2-Clause "Simplified" License
63 stars 4 forks source link

Can't call start_workers more than once #7

Closed juledwar closed 3 years ago

juledwar commented 3 years ago

I want to have worker threads for different queues all running in the same process. To do this I did something like:

engine.start_workers(number=1, block=False)
engine.start_workers(number=1, queue='my-special-queue', block=False)

However the second call is rejected with RuntimeError('Workers are already running')

Is this something that I can fix or is it entirely intentional to block further workers on different queues for some reason I can't see?

juledwar commented 3 years ago

It looks like I should just create a new Engine per queue type with the current code.

juledwar commented 3 years ago

Also, given that I am using the flask_spinach extension this doesn't look particularly straightforward!

NicolasLM commented 3 years ago

Yes, not being able to call start_workers multiple times on the same engine is intentional. The engine holds a reference to one pool of workers.

As you figured, using bare Spinach it is possible to create two brokers and two engines in the same process but the Flask/Spinach integration assumes one Flask app has one Spinach engine, so it will be tough.

juledwar commented 3 years ago

Perhaps we could allow mixing of queue names in the initial call? Although from eyeballing the code that seems rather difficult also as all the Workers share the same queue name.

NicolasLM commented 3 years ago

Closing this one as the design is intentional. Feel free to open another issue to discuss improving the flexibility of queues, if needed.

juledwar commented 3 years ago

That's fair. I did work around it by avoiding the existing Flask extension, which I think is where someone should look first if they want to fix this.