Adding queueing and batching while running multiple dags.

[kriyanshii]

I want to add additional status as in queue. As we are using dagu for running multiple dags at the same time consuming 5min or more time. So what I wanted to do is, queue the dag processes. and it will run one by one. Do you have any insights on this?

[yohamta]

Currently Dagu doesn't have queue system. But we can simulate the same effect by using multiple DAGs.

For Task1, which we aim to run every minute, we start by checking the queue state and proceed only if there are messages waiting:

# run every minute
schedule: "* * * * *"

steps:
  # check queue state
  # exit if there's no message
  - name: check
    # some work to do

To queue tasks for Task1, we can set up another DAG (let's call it Task2) that allows us to push messages into the queue, perhaps through a REST API or other methods. This way, Task2 acts as a trigger for Task1:

Task2

params: "MSG"
steps:
  #  push message into the queue
  - name: push
    command: bash
    script: |
      # push message to the queue (database, redis, file, etc)
      echo $MSG > /tmp/task1-queue

I think implementing queueing feature in Dagu a bit tricky since it doesn't have a central database right now. I'd be happy to hear if there's a good idea for implementing queueing in Dagu.