search

tobymao / saq

Simple Async Queues
https://saq-py.readthedocs.io/en/latest/
MIT License
532 stars 37 forks source link

Support dependencies among jobs #96

Closed greefea closed 12 months ago

greefea commented 12 months ago

What is the best way to do job dependencies with saq, like in rq? In rq, users can specify a depends_on argument when creating a job, and only enqueue the job if the dependencies are finished. I had thought I can hack the before_enqueue and/or before_process hook for this, but it looks like the job will be enqueued/processed no matter what happens in the hook. If anyone can help point me to a good way to support it with saq, I can write the code and do a PR.

BTW, thanks for a fast, light-weighted, yet powerful package!

tobymao commented 12 months ago

depends_on / DAG is something i've explicitly not added due to the complexity involved. i'd be open to a very light weight solution for it though. you probably need to do modify def enqueue to check for dependent jobs,

if dependent jobs are active, or queued, they cannot be enqueued, otherwise they can. you'll also need to make sure they don't start processing.

tobymao commented 12 months ago

happy to chat about it on slack https://tobikodata.com/slack