search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
36.52k stars 14.14k forks source link

Identify duplicate kubernetes section airflow configuration and mark them as deprecated #36389

Open dirrao opened 9 months ago

dirrao commented 9 months ago

Apache Airflow Provider(s)

cncf-kubernetes

Versions of Apache Airflow Providers

apache-airflow-providers-cncf-kubernetes: 7.11.0

Apache Airflow version

2.8.0

Operating System

Cent OS

Deployment

Other

Deployment details

Terraform based deployment

What happened

As we know the pod_template_file file is mandatory from airflow 2 onwards. Users have the provision to provide the default docker image, namespace, etc. in the default pod_template_file. However, we have worker_container_repositor, worker_container_tag, namespace, etc. in the Kubernetes section airflow configuration. It makes sense to mark them as deprecated and remove them eventually.

What you think should happen instead

Configuration fields like worker_container_repositor, worker_container_tag, namespace, etc. have been there since Airflow 1. We can configure the same through the default pod_template_file. So, we can mark them as deprecated and remove them eventually.

How to reproduce

N/A

Anything else

No response

Are you willing to submit PR?

Code of Conduct

eladkal commented 6 months ago

@dirrao are you working on this issue?