Open sonots opened 5 years ago
Maybe, the design will be:
workflowA.dig
_export:
queue: A
workflowB.dig
_export:
queue: B
digdag server (agent) dequeues only from the specified queue:
digdag server --queue A
digdag server --queue B
In addition, supporting priority queues is nice to have.
Maybe, This issue is similar to #727 and #521.
SQS usage is one of other candidates.
I wish that digdag supports multiple queues.
In my company, we setup an AWS autoscaling cluster to run digdag agents as https://techblog.zozo.com/entry/digdag_ha (Japanese) describes.
We want to separate and create multiple clusters to run workflows, for example:
with infrastructure architecture as:
Currently, to achieve similar goal, we have to separate digdag projects, and create another digdag cluster (and web UI)