search

epoch8 / airflow-exporter

Airflow plugin to export dag and task based metrics to Prometheus.
Other
240 stars 75 forks source link

Plugin broken with Airflow 1.10.14 #85

Closed Alien2150 closed 3 years ago

Alien2150 commented 3 years ago

Hey,

this plugin does not seem to be compatible with Airflow 1.10.14. As far as I get they changed the way plugins are loaded: https://github.com/apache/airflow/pull/12859/files#diff-7bfb481064cd476fafe05437d31ba297c9255c13cac38d6f2d2aaeed64af7e13R160

On startup this message is generated: 

[2020-12-11 14:12:29,757] {plugins_manager.py:159} ERROR - Failed to import plugin AirflowPrometheus                                                                            │
│ Traceback (most recent call last):                                                                                                                                              │
│   File "/home/airflow/.local/lib/python3.8/site-packages/airflow/plugins_manager.py", line 150, in load_entrypoint_plugins                                                      │
│     plugin_obj.__usable_import_name = entry_point.module                                                                                                                        │
│ AttributeError: 'EntryPoint' object has no attribute 'module'
Alien2150 commented 3 years ago

Also might be related: https://github.com/apache/airflow/blob/e5ce657c5ff8288c84921d705aab05d6b4028f1e/docs/apache-airflow/upgrading-to-2.rst#id21 as 1.10.14 is a "bridge release" : https://github.com/apache/airflow/blob/e5ce657c5ff8288c84921d705aab05d6b4028f1e/docs/apache-airflow/upgrading-to-2.rst#step-2-upgrade-to-airflow-1-10-14-a-k-a-our-bridge-release

Alien2150 commented 3 years ago

Turns out that it's an issue on Airflow itself (in contact with them) when using Python 3.8+

elephantum commented 3 years ago

Thanks! Keep us informed!

We'll try to implement testing with recent Airflow versions in the meantime.

Alien2150 commented 3 years ago

Ok found a solution. Kudos to the Airflow team. Add "importlib-metadata~=2.0" to requirements or run pip install with python 3.8

potiuk commented 3 years ago

Thanks for reporting @Alien2150!

Yeah For now just installing importib-metadata solves it (and it's only issue for python 3.8). We will decide if it needs a separate release or if just documenting this solution is enough!

kaxil commented 3 years ago

Apologies for the goof-up and thanks for reporting quick @Alien2150 . I tweeted out too https://twitter.com/ApacheAirflow/status/1337415983447814144

elephantum commented 3 years ago

So, maybe just a note in the README should be enough?

potiuk commented 3 years ago

So, maybe just a note in the README should be enough?

I believe we cannot update README in package once it is uploaded to PyPI.

kaxil commented 3 years ago

Maybe you could add a note in https://github.com/epoch8/airflow-exporter/blob/master/README.md instead

elephantum commented 3 years ago

@kaxil Sure, that's exactly what I meant :)