getindata / kedro-airflow-k8s

Kedro Plugin to support running pipelines on Kubernetes using Airflow.
https://kedro-airflow-k8s.readthedocs.io
Apache License 2.0
29 stars 11 forks source link

Support for custom k8s templates #108

Closed mjedrasz closed 3 years ago

mjedrasz commented 3 years ago

This PR adds support for custom Kubernetes templates.

When dealing with complex pod requirements, i.e. many init containers, annotations, volumes, custom env etc., it's easier just to pass a custom template as as string than passing some attributes (currently customising pod template is very limited anyway).

Custom pod templates borrows from resources, i.e. if you specify a __default__ pod template it will be applied to all nodes. Otherwise, only to nodes tagged with a specific value, e.g. k8s_template:spark. The value of the template is F-stringed, which allows you to include some dynamic content, e.g.

type: Pod
  metadata:
    name: {PodGenerator.make_unique_pod_id('{{ task_instance.task_id }}')}

Custom pod templates also let you override the image - it's often the case that more complex containers require heavy images and you don't want all nodes to pay this penalty.


Keep in mind:

mjedrasz commented 3 years ago

Fixed all reasonable codeclimate issues. I wouldn't like to be bothered with the remaining one.

mjedrasz commented 3 years ago

docs added