search

MemVerge / splash

Splash, a flexible Spark shuffle manager that supports user-defined storage backends for shuffle data storage and exchange
Apache License 2.0
127 stars 29 forks source link

java.lang.ClassNotFoundException: org.apache.spark.shuffle.SplashShuffleManager #66

Open Roy-Rupak opened 3 years ago

Roy-Rupak commented 3 years ago

Hello, I have built and set up splash and spark according to the documentation. But when I try to run any spark application I am getting the following error message: java.lang.ClassNotFoundException: org.apache.spark.shuffle.SplashShuffleManager image

This is my spark-default.con file:

spark.files jars/splash-0.7.0.jar,jars/splash-0.7.0-sources.jar,jars/splash-0.7.0-shaded.jar,jars/splash-0.7.0-javadoc.jar spark.driver.extraClassPath ./splash-0.7.0.jar:./splash-0.7.0-sources.jar:./splash-0.7.0-shaded.jar:./splash-0.7.0-javadoc.jar spark.executor.extraClassPath jars/splash-0.7.0.jar,jars/splash-0.7.0-sources.jar,jars/splash-0.7.0-shaded.jar,jars/splash-0.7.0-javadoc.jar

spark.shuffle.manager org.apache.spark.shuffle.SplashShuffleManager

set shuffle manager and storage plugin

spark.shuffle.manager org.apache.spark.shuffle.SplashShuffleManager spark.shuffle.splash.storageFactory com.memverge.splash.shared.SharedFSFactory spark.shuffle.splash.clearShuffleOutput false

set the location of your shared folder

spark.shuffle.splash.folder /home/ruroy/spark/cache

If you could provide any hint to solve this issue that will be very helpful.

Roy-Rupak commented 3 years ago

@jealous Can you please have a look at the issue?

sheperdh commented 3 years ago

@Roy-Rupak It seems you are using Hibench to run spark workload. Can you provide us your spark.conf under Hibench/conf?

Roy-Rupak commented 3 years ago

Hello, @sheperdh thanks for the reply. Here are my spark.conf and hibench.conf files: spark_conf.txt hibench_conf.txt

sheperdh commented 3 years ago

I suspect the root cause is below configurations: spark.driver.extraClassPath ./splash-0.7.0.jar:./splash-0.7.0-sources.jar:./splash-0.7.0-shaded.jar:./splash-0.7.0-javadoc.jar spark.executor.extraClassPath jars/splash-0.7.0.jar,jars/splash-0.7.0-sources.jar,jars/splash-0.7.0-shaded.jar,jars/splash-0.7.0-javadoc.jar Can you try to use absolute file path?