Apache Spark enhanced with native Kubernetes scheduler back-end: NOTE this repository is being ARCHIVED as all new development for the kubernetes scheduler back-end is now on https://github.com/apache/spark/
Possible integration points might be manipulating Pod Disruption Budget for executors, or using anti-affinity to minimize probability of getting targeted based on node overloading.
It is early days with this, but the Rescheduler appears to be another thing that could kill executors: https://docs.google.com/document/d/1KXw02Q0cOF1MUrdpPNiug0yGZlixvPg2SwBycrT5DkE/edit https://github.com/kubernetes/kubernetes/issues/12140
Possible integration points might be manipulating Pod Disruption Budget for executors, or using anti-affinity to minimize probability of getting targeted based on node overloading.
For now, this issue is a vehicle for discussion