search

apache-spark-on-k8s / spark

Apache Spark enhanced with native Kubernetes scheduler back-end: NOTE this repository is being ARCHIVED as all new development for the kubernetes scheduler back-end is now on https://github.com/apache/spark/
https://spark.apache.org/
Apache License 2.0
612 stars 118 forks source link

Mount volumes on driver pod? #630

Closed hochuenw closed 6 years ago

hochuenw commented 6 years ago

I am doing this because I want to store the output back to my local file system since in the cluster mode, the output is written back the driver pod. Is this supported?

mccheah commented 6 years ago

I'm pretty sure we will support volume mounts but only on Spark 2.4. You can also take the tip of the master branch of Spark to get volume support.

Also, please note that we are no longer improving this fork at all and all further development is being done towards apache/master.

mccheah commented 6 years ago

Actually volume support is in the works - https://github.com/apache/spark/pull/21260