Closed khaxis closed 4 years ago
This is likely most promising way to customize worker environment, such as libraries, RAM, GPU, etc. support.
Idea behind Kubernetes: a worker will create a new job in kubernetes with custom docker image. Similar things have been done in airflow's Kubernetes Operator https://kubernetes.io/blog/2018/06/28/airflow-on-kubernetes-part-1-a-different-kind-of-operator/ and in kubeflow pipelines
added in 1.2.4
khaxis
commented
4 years ago khaxis
commented
4 years ago
This is likely most promising way to customize worker environment, such as libraries, RAM, GPU, etc. support.
Idea behind Kubernetes: a worker will create a new job in kubernetes with custom docker image. Similar things have been done in airflow's Kubernetes Operator https://kubernetes.io/blog/2018/06/28/airflow-on-kubernetes-part-1-a-different-kind-of-operator/ and in kubeflow pipelines