search

apache-spark-on-k8s / spark

Apache Spark enhanced with native Kubernetes scheduler back-end: NOTE this repository is being ARCHIVED as all new development for the kubernetes scheduler back-end is now on https://github.com/apache/spark/
https://spark.apache.org/
Apache License 2.0
612 stars 118 forks source link

Allow number of executor cores to have fractional values #361

Closed liyinan926 closed 7 years ago

liyinan926 commented 7 years ago

This commit tries to solve issue #359 by allowing the spark.executor.cores configuration key to take fractional values, e.g., 0.5 or 1.5. The value is used to specify the cpu request when creating the executor pods, which is allowed to be fractional by Kubernetes. When the value is passed to the executor process through the environment variable SPARK_EXECUTOR_CORES, the value is rounded up to the closest integer as expected by the CoarseGrainedExecutorBackend. I expect most use cases of this are to use a fractional of 1 CPU core, e.g., 0.1 or 0.5.

Signed-off-by: Yinan Li ynli@google.com

foxish commented 7 years ago

LGTM except for that one comment. Thanks @liyinan926

liyinan926 commented 7 years ago

Thanks @foxish for reviewing!