rqueue workflow discussion

[XavierMoon]

Is your feature request related to a problem? Please describe. No

Describe the solution you'd like

I'd like to inquire about the idea of adding a workflow to rqueue.

The workflow scenario is as follows:

Job A -> Job B -> Job C: Execute A, B, C in sequence.

Job A (A1, A2, A3) -> Job B: Execute A1, A2, A3 in parallel. Job B waits for Job A (A1, A2, A3) to finish before starting execution.

[sonus21]

What is your case ? Airflow dag has such features. Currently RQueue does not support such features. But sequentially model can be supported

[XavierMoon]

Similar to the second scenario I mentioned, my asynchronous task, Job B, relies on the successful completion of all parallel tasks of Job A (A1, A2, A3) before it can execute. I am curious to know if it's possible to implement my idea using middleware in rqueue.

[XavierMoon]

I can break down the Job A tasks into A1, A2, A3 at the business logic level, but I'm unsure whether this type of workflow with task dependencies (Job A -> Job B) can be implemented within rqueue.

[sonus21]

You're pretty much close to the solution. I can think of solving these two use cases in two ways 1) wherever I've dependent task I can use post processing handler that will enqueue the next task. Using this we can model A -> B

2) wherever I've multiple tasks to complete i can use middleware along with thread pool, but the challenge here is retry on failure. Let's take an example, A has three sub tasks A1, A2 and A3. While running these tasks in parallel, any one of them could fail, on which task failure I should retry ?

There's a way to attach multiple listeners on the same message like in this case we can add three handlers and I'll designated one of them as primary. But now we've to handle race condition for example primary task could complete before other two tasks that's undesirable behaviour. So modelling based upon subtask is tricky at this moment.