This is a fork of the Apache Geode spark connector from tags/rel/v1.1.1 - the intent is to update it to a current version of scala and geode / gemfire. This project is open source and if you would like to contribute feel free to open a pull request.
Apache License 2.0
3
stars
7
forks
source link
saveToGeode("region-name") Task not serializable #1
`scala> distData.saveToGeode("str_str_region")
org.apache.spark.SparkException: Task not serializable
at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:345)
at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:335)
at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:159)
at org.apache.spark.SparkContext.clean(SparkContext.scala:2292)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2022)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2048)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2080)
at org.apache.geode.spark.connector.GeodePairRDDFunctions.saveToGeode(GeodePairRDDFunctions.scala:49)
... 51 elided
Caused by: java.io.NotSerializableException: org.apache.log4j.Logger
Serialization stack:
object not serializable (class: org.apache.log4j.Logger, value: org.apache.log4j.Logger@55105271)
field (class: org.apache.geode.spark.connector.internal.rdd.GeodePairRDDWriter, name: LOG, type: class org.apache.log4j.Logger)
field (class: org.apache.geode.spark.connector.GeodePairRDDFunctions$$anonfun$saveToGeode$1, name: writer$1, type: class org.apache.geode.spark.connector.internal.rdd.GeodePairRDDWriter)
object (class org.apache.geode.spark.connector.GeodePairRDDFunctions$$anonfun$saveToGeode$1, )
at org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:40)
at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:46)
at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:100)
at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:342)
... 58 more`
`scala> distData.saveToGeode("str_str_region") org.apache.spark.SparkException: Task not serializable at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:345) at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:335) at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:159) at org.apache.spark.SparkContext.clean(SparkContext.scala:2292) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2022) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2048) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2080) at org.apache.geode.spark.connector.GeodePairRDDFunctions.saveToGeode(GeodePairRDDFunctions.scala:49) ... 51 elided Caused by: java.io.NotSerializableException: org.apache.log4j.Logger Serialization stack: