search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
36.01k stars 13.98k forks source link

Playback of the workflow statuses in the Graph View for a specific DAG run #12965

Open HaloKo4 opened 3 years ago

HaloKo4 commented 3 years ago

Description

I find myself a lot with the need to see how execution happened in retrospect. I'm looking for a play button that will show how the ETL progressed over time. The information for that play is recorded in the database as for specific run Airflow knows the timestamps of each task try (start, end, status)

So lets say I have ETL that looks like:

A = DummyOperator(task_id='1')
B = PythonOperator(task_id='2')
C = DummyOperator(task_id='3')
  A >> B
  C

There are many possible scenarios for the execution of this ETL in a specific run. C can run before A, they can run in parallel etc.

So what I want to see is the reproduce step according to the timestamps in a graphic way For example (steps):

A becomes brown (Scheduled)
    C becomes grey (Scheduled)
    A becomes grey (Queued)
         A becomes light green (Running)
             C becomes grey (Queued)
                A becomes yellow (Up for retry)
                C becomes light green (Running)
                     C becomes green (Success)
                          A becomes light green (Running)
                              A becomes green (Success)

This representation in text doesn't really tell the full story but it's the best I can do in "tree" manner. I know we have the Gant View but this is a static view and also it shows only the last run of each task.

More Info While playing the playback if I can also pause it or control over the playback speed it would be great. The playback doesn't really execute anything. It's just a nice playback of how the tasks statuses changed over time in a specific DAG run.

bbovenzi commented 3 years ago

I agree. This would be a very useful feature. The Gantt chart is underutilized and this could be a great way to make it more central to the app as the tree and graph views are. I believe that only the task instance history would require backend changes.