Open qiuxin2012 opened 5 years ago
I'm getting the same error on a custom VGG distributed implementation. However, I was able to work around it by setting one more spark configuration parameter.
My command that generated the error was something like:
spark-submit --master yarn --deploy-mode client --driver-memory 80g --executor-memory 60g --num-executors 4 --executor-cores 16 --class com.intel.analytics.bigdl.models.vgg.TrainVGG ${BigDL_HOME}/dist/li$......
To work around I just added --conf "spark.serializer=org.apache.spark.serializer.JavaSerializer"
Thus, the command became:
spark-submit --master yarn --deploy-mode client --driver-memory 80g --executor-memory 60g --num-executors 4 --executor-cores 16 --conf "spark.serializer=org.apache.spark.serializer.JavaSerializer" --class com.intel.analytics.bigdl.models.vgg.TrainVGG ${BigDL_HOME}/dist/li$......
From what I found on other web posts, this error might have something to do with the spark that is configured to not allow more than one existing spark context at a time. Then, with this conf parameter you basically tell Spark that it is ok to create more than one context.
Still, if anyone knows the way to get regular command w/o serializer to work, please share with us!
Hope this helps!
@Bfzanchetta Thanks, --conf "spark.serializer=org.apache.spark.serializer.JavaSerializer"
works fine.
Using kryo deser, will get following error. It's very hard to find out the root cause is kryo, can we add more informantions or fix this bug.