mage-ai / mage-ai

🧙 Build, run, and manage data pipelines for integrating and transforming data.
https://www.mage.ai/
Apache License 2.0
7.39k stars 675 forks source link

Support pod annotations in k8s_executor_config #5204

Open MartinLoeper opened 1 week ago

MartinLoeper commented 1 week ago

Is your feature request related to a problem? Please describe. We would like to add the karpenter annotation karpenter.sh/do-not-disrupt to all pods started by the k8s pipeline execution.

Describe the solution you'd like It would be great to add a config key k8s_executor_config.pod.annotations to metadata.yaml, analogous to: https://github.com/mage-ai/mage-ai/pull/4407

Describe alternatives you've considered We are currently using karpenter node-level annotations which is subpar, as pods are not cleaned consolidated as efficiently.