search

apache-spark-on-k8s / spark

Apache Spark enhanced with native Kubernetes scheduler back-end: NOTE this repository is being ARCHIVED as all new development for the kubernetes scheduler back-end is now on https://github.com/apache/spark/
https://spark.apache.org/
Apache License 2.0
612 stars 118 forks source link

Additional config settings for the memory overhead factor #481

Open ash211 opened 7 years ago

ash211 commented 7 years ago

I'm seeing the default value of 0.10 fail for even reasonably-sized shuffle jobs so expect this value to require some tuning to reliably succeed.

We copied this default value from YARN but it appears that kubernetes is more strict on enforcing memory limits on containers than YARN has been: I have two identically configured clusters of five AWS r3.4xls, one running YARN and the other running kubernetes, with identical driver/executor settings, running identical jobs, and the YARN job succeeds whereas the k8s job fails due to the pod exceeding its memory limit.

foxish commented 7 years ago

To be consistent with YARN, maybe we should do memoryOverhead instead. The memory factor would make the computed value depend on another argument, driverMemory, which isn't ideal IMO.