search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
36.33k stars 14.1k forks source link

Airflow is loading plugins twice #32377

Closed iJanki-gr closed 1 year ago

iJanki-gr commented 1 year ago

Apache Airflow version

2.6.2

What happened

I'm installing a python package that contains airflow.plugins. For some reason when I run airflow plugins the plugins show duplicated twice. This causes an issue loading blueprints:

ValueError: The name 'edit_run_conf' is already registered for this blueprint. Use 'name=' to provide a unique name.

Plugins are definitely being loaded only from python package. The plugins_dir is empty. Verified this also with DEBUG log.

[2023-07-05T14:14:56.911+0000] {plugins_manager.py:300} DEBUG - Loading plugins [2023-07-05T14:14:56.911+0000] {plugins_manager.py:244} DEBUG - Loading plugins from directory: /airflow/plugins [2023-07-05T14:14:56.912+0000] {plugins_manager.py:224} DEBUG - Loading plugins from entrypoints [2023-07-05T14:14:56.920+0000] {plugins_manager.py:227} DEBUG - Importing entry_point plugin cron_display_timezone_plugin [2023-07-05T14:14:56.921+0000] {plugins_manager.py:227} DEBUG - Importing entry_point plugin edit_runconf_plugin [2023-07-05T14:14:56.992+0000] {plugins_manager.py:227} DEBUG - Importing entry_point plugin cron_display_timezone_plugin [2023-07-05T14:14:56.993+0000] {plugins_manager.py:227} DEBUG - Importing entry_point plugin edit_runconf_plugin

Here is how the entrypoints look like in the python package: ./venv/lib/python3.9/site-packages/gr_airflow-0.0.3.dist-info/entry_points.txt

[airflow.plugins]
cron_display_timezone_plugin = gr_airflow.plugins.timezonetrigger.plugin:CronDisplayTimezonePlugin
edit_runconf_plugin = gr_airflow.plugins.edit_runconf.plugin:EditRunConfPlugin

Any one else having this issue?

What you think should happen instead

No response

How to reproduce

Install a package that contains plugin entry points in pyproject.toml

[project.entry-points."airflow.plugins"]
cron_display_timezone_plugin = "gr_airflow.plugins.timezonetrigger.plugin:CronDisplayTimezonePlugin"
edit_runconf_plugin = "gr_airflow.plugins.edit_runconf.plugin:EditRunConfPlugin"

Operating System

Centos 8

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

Code of Conduct

iJanki-gr commented 1 year ago

It must be something to do with python env. I have this issue in python 3.9.2, but it seems all fine in 3.9.17

Airflow is just using the method

importlib_metadata.distributions()

So it is a python issue or something odd in my env.