Read classpath entries from SPARK_EXTRA_CLASSPATH on executors.

apache-spark-on-k8s / spark

Apache Spark enhanced with native Kubernetes scheduler back-end: NOTE this repository is being ARCHIVED as all new development for the kubernetes scheduler back-end is now on https://github.com/apache/spark/

https://spark.apache.org/

Apache License 2.0

612 stars 118 forks source link

Read classpath entries from SPARK_EXTRA_CLASSPATH on executors. #383

Closed mccheah closed 7 years ago

mccheah commented 7 years ago

This makes executors consistent with the driver. Note that SPARK_EXTRA_CLASSPATH isn't set anywhere by Spark itself, but it's primarily meant to be set by images that inherit from the base driver/executor images.

kimoonkim commented 7 years ago

+1. I ran into this issue the other day when I put /etc/hadoop/conf in a custom docker images and wanted put the dir in the classpath. Thanks for fixing this!

mccheah commented 7 years ago

SPARK_EXECUTOR_EXTRA_CLASSPATH isn't reliable anyways because the scheduler will force the environment variable to be set to whatever the user has specified as spark.executor.extraClassPath.