search

databand-ai / airflow-dashboards

Grafana dashboards and StatsD exporter config for Airflow monitoring
Apache License 2.0
266 stars 104 forks source link

Variables in grafana #3

Closed jcdec closed 3 years ago

jcdec commented 3 years ago

Hello Rozhkov,

I'm using your Aiflow monitoring project with Grafana, but I have a big problem. In my environment, we have a lot of aiflow dags. How did you manage to filter the dag_id variable in grafana?

See the image below for two examples of different dag. image

rozhok commented 3 years ago

Hello, @jcdec! For per-dag view, you should use "Airflow DAG Dashboard". Checkout this dashboard https://github.com/databand-ai/airflow-dashboards/blob/main/grafana/dag-dashboard.json which provides exactly what you need.

If you've configured dashboard by yourself, make sure you have proper StatsD mappings configured. You can extract your dag_id from metric by using regex:

  - match: "*.dagrun.duration.success.*"
    match_metric_type: observer
    name: "af_agg_dagrun_duration_success"
    labels:
      airflow_id: "$1"
      dag_id: "$2"```

Check out our article [Monitoring Airflow with Prometheus, StatsD and Grafana])https://databand.ai/blog/everyday-data-engineering-monitoring-airflow-with-prometheus-statsd-and-grafana/) for details.