search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
37.12k stars 14.3k forks source link

KubernetesPodOperator annotations kwarg should support templating. #10270

Closed iantbutler01 closed 1 year ago

iantbutler01 commented 4 years ago

Description

I would like the annotations kwarg in the KubernetesPodOperator to support templating.

Use case / motivation

With how the KubernetesExecutor runs tasks right now I can't generate an id in my dag file and use it for each task in the dag because it will be regenerated for each subsequent task. When the dag files are copied and ran on the individual Kubernetes pod the generation step is rerun.

For example if I set a variable x = str(uuid.uuid4()) at the beginning of my dag file and then pass x as an arg to 3 tasks each task will have a different value for that parameter on the KubernetesExecutor. To get around that we are using an external metadata DB and xcoms.

We would love to be able to template xcom data into the pod annotations.

Our specific use case is we're doing versioned web crawls using Airflow as our job scheduler for crawling and loading data in spark and we use annotations on the pod and xcoms to know which pod to communicate with over HTTP from different tasks for various monitoring and management tasks.

Related Issues

I have not seen one.

mik-laj commented 4 years ago

@iantbutler01 It makes sense. Wouldn't you like to prepare a PR? I am happy to help with the review and merging the change.

iantbutler01 commented 4 years ago

Yes I am happy to work on this!

mik-laj commented 4 years ago

Great! I assigned you to this ticket. 😺

eladkal commented 3 years ago

@iantbutler01 are you still working on it?

iantbutler01 commented 3 years ago

I am not, my work has carried me far away from Airflow and I do not have the time these days @eladkal