search

airflow-helm / charts

The User-Community Airflow Helm Chart is the standard way to deploy Apache Airflow on Kubernetes with Helm. Originally created in 2017, it has since helped thousands of companies create production-ready deployments of Airflow on Kubernetes.
https://github.com/airflow-helm/charts/tree/main/charts/airflow
Apache License 2.0
630 stars 474 forks source link

Add support for standalone DagProcessorManager #731

Closed Linux-oiD closed 11 months ago

Linux-oiD commented 1 year ago

Checks

Motivation

Airflow community in version 2.3.0 introduced standalone DagProcessorManager. https://airflow.apache.org/blog/airflow-2.3.0/#dagprocessormanager-as-standalone-process-aip-43 Would be great to have this feature supported in chart. Official Airflow helmchart provide basic support of it.

Implementation

The standalone DagProcessor has a CLI argument to specify subfolder of AIRFLOW_HOME folder to parse: https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#dag-processor

Would be great to have possibility to provide map of list of several folders with dags and deploy separate instance/deployment of dag-processor-manager for each of folders.

Are you willing & able to help?

Linux-oiD commented 1 year ago

It was also mentioned here: https://github.com/airflow-helm/charts/issues/572#issuecomment-1119467638

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had activity in 60 days. It will be closed in 7 days if no further activity occurs.

Thank you for your contributions.

Issues never become stale if any of the following is true:

  1. they are added to a Project
  2. they are added to a Milestone
  3. they have the lifecycle/frozen label