[DCOS-45376] Add Spark-CLI support for `--jars` flag

samvantran commented 5 years ago

What changes were proposed in this pull request?

This updates the Spark CLI to support the --jars flag and was inspired by this workaround.

Note: this also fixes a bug in the CLI where passing --jars would result in incorrect parsing of submit-args and the command would fail to submit. This should be backported to both 2.4.0 and 2.5.0

How were these changes tested?

Added unit tests to verify additional --conf flags are added as part of submit-args
Added an integration test to submit a job with --jars with an app missing a Java class
Manually verified spark jobs complete on a strict cluster with a new stub

Release Notes

Added Spark-CLI support for --jars
Fixed a bug where passing --jars would incorrectly parse and fail to submit

samvantran commented 5 years ago

Unfortunately @akirillov mapping --jars <urls> to --conf spark.jars=<urls> does not appear to work. Seems like the extra jar was not downloaded nor added to the class path. You can see in this test run I tried (note Class path entries is empty) and the job failed to find the class I specified.

I think spark.jars.packages works bc Spark source will attempt to search in maven central repos whereas spark.jars does not seem to work as advertised. Perhaps we'll need to stick with the workaround.

samvantran commented 5 years ago

Thanks @akirillov. Yes this is for a bug fix - we can explore fixing spark source code in the future since we discovered spark.jars is blacklisted from conf creation

mesosphere / spark-build