search

etsy / boundary-layer

Builds Airflow DAGs from configuration files. Powers all DAGs on the Etsy Data Platform
Apache License 2.0
262 stars 58 forks source link

Fix KubernetesPodOperator's broken resources arg #118

Closed eap closed 2 years ago

eap commented 3 years ago

Since at least airflow version 1.10.4 the kubernetes pod operator resources argument has taken a dictionary rather than the actual resources object. reference docs. This change updates the config to properly set resource constraints.

Without this change, users of the Kubernetes Pod Operator will get the following error in the generated airflow;

TypeError: airflow.kubernetes.pod.Resources() argument after ** must be a mapping, not Resources
peleyal commented 3 years ago

@vchiapaikeo Can you (or someone else in the team) please take a look at this PR? We will be moving to our own plugin, however there are fixes here and there to the current configuration. PTAL, thanks!

vchiapaikeo commented 3 years ago

@peleyal , @eap, since we are still on version 1.10.3, we'd like to leave this as is. We are in fact still passing resources through as a dictionary. However, you can make this change to your plugin version and still be able to use it (you'd create an identical version of this config file and it will be prioritized over the boundary-layer version).

peleyal commented 3 years ago

Thanks @vchiapaikeo, I finally found some time to create our own plugin. Your documentation in #114 was really useful. Thank you!

eap commented 2 years ago

since we manage our own plugin and this is incompatible with the default plugin. I'm closing this PR