search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
37.02k stars 14.28k forks source link

Support different plugins_folder setting in helm chart when airflow version > 2.0.0 #39562

Open tankim opened 6 months ago

tankim commented 6 months ago

Description

When configuring Airflow using a Helm chart, the core section allows you to set the plugins_folder. This setting is applied equally across all web servers and schedulers. Many users utilize git-sync, and it's common for the plugins' path to be located within a git-synced repository.

With Airflow version 2.0.0 and above, Dag Serialization is enabled by default, which means that the web server cannot use a git-sync sidecar. This is generally not an issue when developing custom operators, but it can cause confusion if you want to customize the web UI and the plugins folder on the web server is not synced through git-sync.

Therefore, it would be beneficial to guide the explicit setting of the plugins_folder in the Helm chart, allowing separate configurations for the web server and scheduler. This approach would enable more flexible management of each component's settings, allowing finer adjustments to meet specific requirements.

Use case/motivation

When attempting to customize the Airflow Web UI based on AirflowPlugin, I followed the guidance to write scripts in the plugins directory. However, with Airflow version 2.0.0 and above, there was confusion in using the webserver plugins because git-sync does not operate as a sidecar for the webserver.

As documented at Airflow's Helm Chart Guide, it is generally not recommended to use git-sync with persistence together. This recommendation has significantly limited the possible workarounds to address this issue.

Related issues

Airflow Slack

Are you willing to submit a PR?

Code of Conduct

boring-cyborg[bot] commented 6 months ago

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

shahar1 commented 3 months ago

@tankim Would you still like to work on this issue? Also, in the "Related Issues" section - I'd suggest to directrly quote from the Airflow slack messages instead of putting a link, otherwise it will be hidden after a while.