apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
36.95k stars 14.26k forks source link

Helm chart does not support multiple executors #43224

Open rcheatham-q opened 1 week ago

rcheatham-q commented 1 week ago

Official Helm Chart version

1.15.0 (latest released)

Apache Airflow version

2.10.2

Kubernetes Version

1.30.4

Helm Chart configuration

executor: CeleryExecutor,KubernetesExecutor

Docker Image customizations

n/a

What happened

First, the JSON schema specification failed as it has an allowlist of all known supported executors. Then, when I skipped schema verification, the deployment failed because the executor value is used as a label on at least one deployment, and commas are not allowed in label values.

❯ echo 'executor: CeleryExecutor,KubernetesExecutor' > values.yaml
❯ helm upgrade --install airflow apache-airflow/airflow --values values.yaml                                
Error: UPGRADE FAILED: values don't meet the specifications of the schema(s) in the following chart(s):
airflow:
- executor: executor must be one of the following: "LocalExecutor", "LocalKubernetesExecutor", "CeleryExecutor", "KubernetesExecutor", "CeleryKubernetesExecutor", "airflow.providers.amazon.aws.executors.batch.AwsBatchExecutor", "airflow.providers.amazon.aws.executors.ecs.AwsEcsExecutor"

❯ helm upgrade --install airflow apache-airflow/airflow --values values.yaml --skip-schema-validation
Release "airflow" does not exist. Installing it now.
Error: 1 error occurred:
    * Deployment.apps "airflow-scheduler" is invalid: metadata.labels: Invalid value: "CeleryExecutor,KubernetesExecutor": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')

What you think should happen instead

The helm chart should deploy successfully when multiple executors are specified as described in the docs.

How to reproduce

echo 'executor: CeleryExecutor,KubernetesExecutor' > values.yaml
helm upgrade --install airflow apache-airflow/airflow --values values.yaml

and

helm upgrade --install airflow apache-airflow/airflow --values values.yaml --skip-schema-validation

Anything else

No response

Are you willing to submit PR?

Code of Conduct

potiuk commented 1 week ago

Sure this is good feature to add. If you would liek to do @rcheatham-q - you can add it, otherwise it will have to wait for someone to implement it.

potiuk commented 1 week ago

Shall I assign you @rcheatham-q ?

rcheatham-q commented 1 week ago

I don't have time to do this right now, but if the opportunity presents itself I'm willing to take this on.

amoghrajesh commented 19 hours ago

@rcheatham-q do you have some time to work on this? Otherwise I can pick this up

rcheatham-q commented 1 minute ago

@amoghrajesh I do not have time to work on this