search

python-arq / arq

Fast job queuing and RPC in python with asyncio and redis.
https://arq-docs.helpmanual.io/
MIT License
2.1k stars 173 forks source link

Use pubsub and blocking for improved response times #404

Open m-kuhn opened 1 year ago

m-kuhn commented 1 year ago

arq currently uses a polling mode (default 0.5 seconds) to fetch jobs on worker side and also to fetch results. I think it should be possible to use blocking requests (on worker side) and pubsub (possibly with keyspace notifications, but not necessarily) for fetching the result.

This should be quite reliable in most scenarios, however it can occur that a job gets lost (there is no guarantee for pubsub delivery after all), so one could still poll from time to time to be on the safe side.

Are there any opinions on development in this direction?

From https://github.com/samuelcolvin/arq/pull/374#issuecomment-1570580555

m-kuhn commented 1 year ago

CC @arturzx

gg718 commented 1 year ago

See also how rq recently implemented this: https://github.com/rq/rq/pull/1939

JonasKs commented 1 year ago

A draft PR welcome.😊

samuelcolvin commented 6 months ago

My plan is to move to redis streams which are by far the best solution for tasks like those performed by arq. Stay tuned for a big update.

samuelcolvin commented 6 months ago

See #437.

Wh1isper commented 3 months ago

I agree that redis stream is the best option right now. Can't wait for the arq update, I developed a redis stream based library, aka brq. If you're in a hurry, you may try it now.