search

big-data-europe / docker-spark

Apache Spark docker image
2.04k stars 698 forks source link

Classpath issue : Could not find or load main class org.apache.spark.launcher.Main #113

Closed majyphi closed 2 years ago

majyphi commented 3 years ago

I'm trying to setup a spark-submit image to launch our jobs. To test the setup I run & enter the latest spark-submit image with : docker run -it bde2020/spark-submit:latest bash Once inside I try to start a spark-shell : /spark/bin/spark-shell

I am greeted with : 

Error: Could not find or load main class org.apache.spark.launcher.Main
/spark/bin/spark-class: line 96: CMD: bad array subscript

Down the rabbit hole this is the line that fails in spark-class : java -Xmx128m -cp '/spark/jars/*' org.apache.spark.launcher.Main org.apache.spark.deploy.SparkSubmit --class org.apache.spark.repl.Main --name 'Spark shell'

It seems the -cp /spark/jars/* bit does not correctly incorporate the jars inside the directory.

Do you have any idea on the cause of this problem ?

EDIT : Running on Red Hat Enterprise Linux Server 7.8 (Maipo) this might be a factor as I have no issues on Ubuntu 20+

majyphi commented 2 years ago

After investigation it was indeed caused by the OS, or the underlying system.