Heavy CPU usage on large number of DAGs

epoch8 / airflow-exporter

Airflow plugin to export dag and task based metrics to Prometheus.

Other

248 stars 75 forks source link

Closed elephantum closed 5 years ago

elephantum commented 5 years ago

As discussed here: https://github.com/epoch8/airflow-exporter/issues/23#issuecomment-441232655 there's an issue with collecting metrics on the database with large number of DAGs.

We need:

d-goldin commented 5 years ago

In our fork, we have applied a fix like: https://github.com/soundcloud/airflow-exporter/blob/d-goldin/disable_auto_discover_names/prometheus_exporter.py#L122 - maybe it's something that could work for you too or potentially be upstreamed. This mitigates the issue of workers running this query, which caused the biggest chunk of load for us. Implementing a description function would still probably be better, but was not overly urgent to us. (CC @ebartels)

d-goldin commented 5 years ago

Seems like it's actually already fixed using describe in 0.5.1, so we will try that one instead.

elephantum commented 5 years ago

I believe this is obsolete. Closing