feat: Add per-task queue routing

[henri42]

Add the possibility to route task individually in a workflow.

example.ETL:
  tasks:
    - [EXTRACT, queue1]
    - [TRANSFORM, queue2]
    - LOAD

In the following case, the task-level queue overrides the workflow-level queue for the task where a queue is defined.

example.ETL:
  tasks:
    - EXTRACT
    - [TRANSFORM, queue2] # task-level
    - LOAD
  queue: queue1 # workflow-level

[CodePint]

related: https://github.com/ovh/celery-director/pull/134

[ncrocfer]

Hi @henri42,

First of all thank you for this PR, it brings an interesting feature in Celery Director ! Unfortunately after a few tests I see it doesn't work for all the cases.

For instance everything is working for a simple ETL example:

But it doesn't work when a group of tasks is executed:

I think you can update the builder to support it, but for me the syntax you choose is a bit confusing. The users will think it's a group of tasks, while it's just an array of 2 elements with the task and its queue.

The best is to provide another syntax, for instance using the queue key:

example.ETL:
  tasks:
    - EXTRACT
    - TRANSFORM
    - LOAD
  queue:
    default: q1
    customs:
      EXTRACT: q2

Note this is just an example, I just wrote it quickly so I'm not sure about the final syntax ;) Also note if you do that you have to continue to support the current syntax, namely:

  queue: q1

is the same as:

  queue:
    default: q1

[henri42]

Thanks for your feedback ! Indeed this syntax can be confusing. I chose it to keep readability but it might be better with the syntax you suggested. I will work on this new syntax and handle task-level queues for the tasks in groups.

[henri42]

Thanks ! That's done.