search

epoch8 / airflow-exporter

Airflow plugin to export dag and task based metrics to Prometheus.
Other
248 stars 75 forks source link

Add airflow_task_last_status metric #97

Closed nvn01234 closed 2 years ago

nvn01234 commented 3 years ago

Similar with airflow_dag_last_status metric, airflow_task_last_status is status of tasks in the last dagrun of each dag. This metric can help to calculate the completion of the last dagrun (example: the dag is running, 50% of tasks are finished)

Result will look like this, for each task we will have 12 statuses: image

elephantum commented 3 years ago

I'm a little bit concerned about that, some of the users complained about large number of metrics, some even forked plugin to remove tasks monitoring.

elephantum commented 3 years ago

What would be a use case to monitor last status of a specific task?

nvn01234 commented 3 years ago

In a failed dagrun, I want to know which tasks are failed in the last dagrun and how many (or the percentage of) failed tasks.

Should I remove other task states so that only failed tasks appeared in the metrics? or only count the number for each states in the last dagrun?

nvn01234 commented 3 years ago

I will submit another PR (or some), including diagrams that I need to build on Grafana, so you can choose the better option.

elephantum commented 2 years ago

Closing as stale