search

epoch8 / airflow-exporter

Airflow plugin to export dag and task based metrics to Prometheus.
Other
247 stars 75 forks source link

export dag/task metrics #49

Open lelvisl opened 5 years ago

lelvisl commented 5 years ago

It may be useful to export XCOM variables for monitoring dag/task progress.

elephantum commented 5 years ago

@lelvisl can you give complete use case for this functionality?

lelvisl commented 5 years ago

I need some opportunity to monitor status of my dugs. Example - how many rows had selected from source. (Some of my dags lasts more then 9 hours). In dag I can update Xcom variable.

(Now I try to use http in python operator & prom push gateway)

lelvisl commented 5 years ago

P.S. Спасибо за очень полезный плагин!

elephantum commented 5 years ago

(я оставлю обсуждение на английском, мало ли кто подключится)

There's a problem with exporting Dag/Task specific metrics to prometheus: Airflow does not guarantee that Task runs at scheduled date (there might be delay between scheduling and running or Task can be backfilled).

So metrics in prometheus would not line up with logical execution date.

We have similar need (to monitor custom metrics from dags) and I was thinking that it might be an extension of Airflow graphing functionality (Landing times, Task duration etc). So different plugin or patch to Airflow.

lelvisl commented 5 years ago

Yes, you right, xcom variables is not the right decision.

As I understood, there is two ways: