search

PBWebMedia / airflow-prometheus-exporter

Export Airflow metrics (from mysql) in prometheus format
MIT License
29 stars 10 forks source link

How to determine the number of tasks currently running? #10

Open estahn opened 6 years ago

estahn commented 6 years ago

Questions:

DemonTPx commented 6 years ago

Determining how many tasks are running is not possible (yet at least...) You can however use the airflow_dag_run_state metric to see how many and when certain events happened (success, failed, running)

I think that task duration would be way too much data to export, since it would need a separate metric/dimension for every task instance. Maybe a different approach to this would be possible by counting the number of seconds taken and the number of tries or something... But it isn't possible at this moment.

estahn commented 6 years ago

@DemonTPx We want to use the metric to scale up pods in Kubernetes. Any idea what metric would be useful for that? We thought something like queue size.

colleym commented 3 years ago

We use the GCP Composer and the Stackdriver Exporter for that metric. But that does not help you, if you are using a native Aiflow deployment. https://cloud.google.com/monitoring/api/metrics_gcp#gcp-composer