search

apache-spark-on-k8s / spark

Apache Spark enhanced with native Kubernetes scheduler back-end: NOTE this repository is being ARCHIVED as all new development for the kubernetes scheduler back-end is now on https://github.com/apache/spark/
https://spark.apache.org/
Apache License 2.0
612 stars 118 forks source link

Specify hostpath volume and mount the volume in Spark driver and executor pods #614

Open sumansomasundar opened 6 years ago

sumansomasundar commented 6 years ago

In some cases (for example, log aggregation) it is desirable to have a hostpath volume mounted in the driver and executor pods. With #38 this will be possible, but I am not sure when it will be available.

So, something similar to #397 will be helpful for this case as well.

What I am proposing is an option to specify the hostpath volume and have it mounted in the driver and executor pods.

sumansomasundar commented 6 years ago

I have made changes that support the following configuration for spark-submit command: --conf spark.kubernetes.driver.hostpath.= If this approach sounds good, I can create a pull request for this change.