search

apache-spark-on-k8s / spark

Apache Spark enhanced with native Kubernetes scheduler back-end: NOTE this repository is being ARCHIVED as all new development for the kubernetes scheduler back-end is now on https://github.com/apache/spark/
https://spark.apache.org/
Apache License 2.0
612 stars 118 forks source link

Tolerations and node affinity support #623

Open Lemmons opened 6 years ago

Lemmons commented 6 years ago

Taints, tolerations, and node affinities seem to be the way to provide fine-grain scheduling support in a kubernetes cluster. Maybe I'm blind, but I can't seem to find any way to provide pod spec level properties when running spark. The closest I can get is spark.kubernetes.node.selector., but this doesn't provide the level of required control.

I can spin up another dedicated kube cluster to only host spark, but I would prefer to instead just have a dedicated node pool, which adding support for tolerations would do.

dawnbreaks commented 5 years ago

+1 Yes, we need this feature too. Is there a plan to support tolerations?

rvesse commented 5 years ago

NB - Per the README all development work and discussion has moved to the main apache/spark repo

You can probably do this with the new Pod Templates feature which is currently only available in master and is targeted for release with Spark 3.0 sometime later in the year