spark-submit (python) reports "Error: Unrecognized option: --driver-class-path="

[shuoyenl]

spark-submit --master nomad:{addr of master} --deploy-mode cluster --conf spark.nomad.sparkDistribution=https://s3.amazonaws.com/nomad-spark/spark-2.1.0-bin-nomad.tgz --py-files {addr of hello.py} {addr of hello.py}

where hello.py just print("hello")

The batch job got scheduled by Nomad but then the container exited with code 1. The logs (nomad logs -stderr) show the following error:

Error: Unrecognized option: --driver-class-path=

Usage: spark-submit [options] <app jar | python file> [app arguments]
Usage: spark-submit --kill [submission ID] --master [spark://...]
Usage: spark-submit --status [submission ID] --master [spark://...]
Usage: spark-submit run-example [options] example-class [example args]

Options:
.......

As a comparison, the java example works with our setup:

spark-submit --master nomad:{addr of master} --deploy-mode cluster --conf spark.nomad.sparkDistribution=https://s3.amazonaws.com/nomad-spark/spark-2.1.0-bin-nomad.tgz --class org.apache.spark.examples.JavaSparkPi https://s3.amazonaws.com/nomad-spark/spark-examples_2.11-2.1.0-SNAPSHOT.jar

[angrycub]

Should anyone else encounter this problem before it is resolved, supplying a concrete value for --driver-class-path (even /dev/null) in the original spark-submit command will workaround this issue.

[angrycub]

Looking at: https://github.com/hashicorp/nomad-spark/blob/f476f50bee1eb49d314ae427bd2155d073275a98/resource-managers/nomad/src/main/scala/org/apache/spark/scheduler/cluster/nomad/DriverTask.scala#L97-L99 It would seem that "--driver-class-path=" should only be emitted when

(additionalJarUrls ++ conf.getOption("spark.driver.extraClassPath")) 
     .map(j => new URI(j).getPath).mkString(":") 

evaluates to a non-empty value.

[barnardb]

Fixed by #10