search

epoch8 / airflow-exporter

Airflow plugin to export dag and task based metrics to Prometheus.
Other
240 stars 75 forks source link

Breaking change in Airflow 1.10.3(rc2) - removal of DagStat class #46

Closed fahran closed 5 years ago

fahran commented 5 years ago

Unfortunately a class that the exporter depends on, DagStat, has been removed in the latest version of Airflow (tested with v1.10.3rc2)

The error as reported in the UI is: "Broken plugin: [/usr/local/airflow/plugins/airflow-exporter-0.5.4/prometheus_exporter.py] cannot import name 'DagStat'"

From the changelog: https://github.com/apache/airflow/blob/master/CHANGELOG.txt, the relevant line is: [AIRFLOW-3573] Remove DagStat table (#4378). That jira ticket is available here: https://issues.apache.org/jira/browse/AIRFLOW-3573, and contains the commit log etc.

Looking at the code, three fields are used from DagStat (id, state & count). Looking at airflow's www/views.py code (and the modification made to it in light of [AIRFLOW-3573], tagged as [AIRFLOW-3561], commit# 0cc078a86d00c0f798b2b26d828126be99af9c6c) it seems like DagRun would work instead. https://github.com/apache/airflow/blob/6970b233964ee254bbb343ed8bdc906c2f7bd974/airflow/www/views.py#L278

Do you guys have any thoughts?

elephantum commented 5 years ago

@fahran thanks for bringing this issue up!

That's a bug that should be fixed.

Seems like DagRun should be a correct replacement for this info.

I'm not sure, that we'll have time for that right now. If you could make a PR - that would be great.

fahran commented 5 years ago

Cheers @elephantum - thanks for your confirmation of the diagnosis too - much better it's been looked at by someone more familiar with Airflow internals that I currently am!

We love the plugin, so if no-one else has got to it in the meantime, we'll put landing a fix for this in our next sprint (~a couple of weeks) :)

sockeye44 commented 5 years ago

Just encountered the same problem, looks like switching to DagRun really solves it. Created PR #47 with a fix

elephantum commented 5 years ago

@fahran for now it's impossible to preserve compatibility with 1.10.3 and previous versions simultaneously. (possible but harder)

I merged @sockeye44 PR into branch airflow-1.10.3 use it to have plugin working.

elephantum commented 5 years ago

Closing this issue for now. Feel free to reopen if necessary.