search

apache-spark-on-k8s / spark

Apache Spark enhanced with native Kubernetes scheduler back-end: NOTE this repository is being ARCHIVED as all new development for the kubernetes scheduler back-end is now on https://github.com/apache/spark/
https://spark.apache.org/
Apache License 2.0
612 stars 118 forks source link

Control executor scaling by external call/signal to a driver #473

Open erikerlandson opened 7 years ago

erikerlandson commented 7 years ago

Currently, the scaling modes supported are: 1) static (set desired executors at startup time) and 2) dynamic allocation (exponential scale-up, adaptive scale-down).

Allowing a driver to be messaged in some way from the outside, to update the number of executors it should run, allows other spark scaling algorithms to be written externally. For example, any user could implement their own custom controller that periodically updates the scale for each spark job running on the cluster. Or a user could be given a simple control to manually scale their cluster up or down.

foxish commented 7 years ago

One thing that's a prerequisite to having external scaling, is to have the spark scheduler backend understand that, and not attempt to manage the lifetimes of any of the executors. Else, I can imagine races where the two schedulers don't see eye to eye.

However, question - are there use-cases where we think the driver makes worse scheduling decisions than a custom controller could?

erikerlandson commented 7 years ago

My intuition is that a custom controller is likely to do better at some use cases, one significant example being: adjusting executor scaling based on quota systems (e.g. the kinds of quota systems intended to be managed via the arbitrator).

IMO, the executor lifetime logic would remain almost exactly as it is now. The main difference would be that instead of consulting whether the task queue is non-empty or empty, to determine how many executors to request, it would take its cue from an externally supplied number.

Depending on use case, this external number might serve as a maximum, instead of a hard number of executors. So that if no executors are idle, it could still scale lower, but would never scale higher.

ash211 commented 7 years ago

@erikerlandson this doesn't seem kubernetes-specific -- I can imagine someone setting up a similar external scaler for Spark apps running in YARN. Maybe it's best to have this discussion in the upstream Apache issue tracker?

foxish commented 7 years ago

There is a k8s component called the Horizontal Pod Autoscaler that can consume arbitrary "metrics" and make scaling decisions. This could be an extension of that if the right metrics can be published and the CRD can expose a scale subresource.

I'm not convinced about concrete uses for K8s-driven control being real yet however. It seems like a "nice to have" at the moment.

On Sep 5, 2017 10:31 PM, "Andrew Ash" notifications@github.com wrote:

@erikerlandson https://github.com/erikerlandson this doesn't seem kubernetes-specific -- I can imagine someone setting up a similar external scaler for Spark apps running in YARN. Maybe it's best to have this discussion in the upstream Apache issue tracker?

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/apache-spark-on-k8s/spark/issues/473#issuecomment-327379171, or mute the thread https://github.com/notifications/unsubscribe-auth/AA3U5-JiztRkTKQQ1GmrK0poh3uwmGFjks5sfi43gaJpZM4PHxUR .