apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
36.26k stars 14.08k forks source link

Not working in Chart multi_namespace_mode_namespace_list is not working. #41636

Open PavelJacoboGonzalez opened 3 weeks ago

PavelJacoboGonzalez commented 3 weeks ago

Official Helm Chart version

1.15.0 (latest released)

Apache Airflow version

2.9.3

Kubernetes Version

Server Version: version.Info{Major:"1", Minor:"30", GitVersion:"v1.30.0", GitCommit:"7c48c2bd72b9bf5c44d21d7338cc7bea77d0ad2a", GitTreeState:"clean", BuildDate:"2024-04-17T17:27:03Z", GoVersion:"go1.22.2", Compiler:"gc", Platform:"linux/amd64"}

Helm Chart configuration

executor: "CeleryKubernetesExecutor"

multiNamespaceMode: true

env:

Docker Image customizations

FROM apache/airflow

USER root RUN apt-get update \ && apt-get install -y --no-install-recommends \ vim \ && apt-get autoremove -yqq --purge \ && apt-get clean \ && rm -rf /var/lib/apt/lists/*

USER airflow

Extends providers

RUN pip install -U --no-cache-dir \ apache-airflow-providers-mongo \ clickhouse-driver \ clickhouse-cityhash lz4 \ airflow-clickhouse-plugin[common.sql]

Extends packages

RUN pip install --no-cache-dir pymongo pika requests selenium lxml

What happened

The configuration for multi_namespace_mode_namespace_list should be enabled, but even I've added namespace variables in env, the configuration is just enabling multi_namespace_mode, but it's not configuring multi_namespace_mode_namespace_list.

$ airflow config list
...
[kubernetes]
airflow_configmap = airflow-config
airflow_local_settings_configmap = airflow-config
multi_namespace_mode = True
namespace = airflow
pod_template_file = /opt/airflow/pod_templates/pod_template_file.yaml
worker_container_repository = airflow_local
worker_container_tag = 1.0.4

multi_namespace_mode_namespace_list is missing even if I set variable in .yaml

What you think should happen instead

there should be multi_namespace_mode_namespace_list in the [kubernetes] section so we can choose in which namespaces we would like the schedule pods.

How to reproduce

enable or set AIRFLOW__KUBERNETES_EXECUTOR__MULTI_NAMESPACE_MODE with a list of specific namespaces. Then try to run a KPO inside a not listed namespace. KPO task should fail. It it runs, it's because multi_namespace_mode_namespace_list is not applied.

Anything else

Thanks you very much, and sorry for my english

Are you willing to submit PR?

Code of Conduct

boring-cyborg[bot] commented 3 weeks ago

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.