Kubernetes task decorator default namespace behaves different than KubernetesPodOperator when in cluster

[ketozhang]

Apache Airflow version

Other Airflow 2 version (please specify below)

If "Other Airflow 2 version" selected, which one?

2.9.2

What happened?

Given these settings for KuberneteseExecutor:

Section	Key	Value
kubernetes_executor	in_cluster	True
kubernetes_executor	namespace	redacted_not_default

the behavior of @task.kubernetes() vs KubernetesOperator() when not setting namespace differs:

• @task.kubernetes() defaults to namespace="default"
• KuberntesPodOperator() defaults to namespace=None which uses the cluster namespace when in_cluster is true.

What you think should happen instead?

KubernetesPodOperator and @task.kubernetes should use the in-cluster namespace by default (i.e., the worker pod's namespace).

How to reproduce

1. Use a namespace that isn't default, let's call it foobar for this example
2. Set the service account to only allow pod API access to foobar namespace.

3. Run this DAG

   with DAG(...):
       @task.kubernetes(image="hello-world"):
       def foo():
           ...
       taskA = foo()
   
       taskB = KubernetesPodOperator(image="hello-world")

You'll see K8s API error for the wrong namespace. In my case, the KPO decorator is trying to access the airflow namespace despite the pod is running in the correct namespace

HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:serviceaccount:airflow:airflow\" cannot list resource \"pods\" in API group \"\" in the namespace \"airflow\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}

Here's the KPO decorator task's pod spec:

metadata:
  annotations:
    dag_id: lensing_model_classification
    run_id: manual__2024-10-11T22:11:31.074729+00:00
    task_id: redacted
    try_number: '4'
  labels:
    airflow-worker: '0'
    airflow_version: 2.9.2
    dag_id: redacted
    kubernetes_executor: 'True'
    run_id: manual__2024-10-11T221131.0747290000-e1a8551c6
    task_id: redacted
    try_number: '4'
  name: redacted
  namespace: 'foobar'
...

Operating System

Amazon Linux 2

Versions of Apache Airflow Providers

apache-airflow-providers-cncf-kubernetes==7.14.0

Deployment

Other 3rd-party Helm chart

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• [ ] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[ketozhang]

In the meantime you can fix this by explicitly setting the namespace to None:

with DAG(...):
  @task.kubernetes(image="hello-world", namespace=None)
  def foo():
      ...