search

HCADatalab / powderkeg

Live-coding the cluster!
Eclipse Public License 1.0
159 stars 23 forks source link

Referencing java.lang.Void/TYPE breaks kryo serialization #16

Closed paul-amperity closed 7 years ago

paul-amperity commented 7 years ago

Repro:

(require '[powderkeg.core :as keg])
(keg/connect! "local")
(def voidType java.lang.Void/TYPE)
(into [] (keg/rdd [1 2 3 4 5 6] (map #(* 2 %))))

Throws exception:

NullPointerException   com.esotericsoftware.kryo.serializers.FieldSerializer.rebuildCachedFields (FieldSerializer.java:66)

com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Unable to create serializer "com.esotericsoftware.kryo.serializers.FieldSerializer" for class: void
Serialization trace:
vars (powderkeg.core$barrier_BANG_$fn__11172$fn__11173)
fn (clojure.lang.Delay)
 at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write (FieldSerializer.java:585)
    com.esotericsoftware.kryo.serializers.FieldSerializer.write (FieldSerializer.java:213)
    com.esotericsoftware.kryo.Kryo.writeObject (Kryo.java:501)
    com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write (FieldSerializer.java:564)
    com.esotericsoftware.kryo.serializers.FieldSerializer.write (FieldSerializer.java:213)
    com.esotericsoftware.kryo.Kryo.writeClassAndObject (Kryo.java:568)
    org.apache.spark.serializer.KryoSerializationStream.writeObject (KryoSerializer.scala:158)
    org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject (TorrentBroadcast.scala:203)
    org.apache.spark.broadcast.TorrentBroadcast.writeBlocks (TorrentBroadcast.scala:102)
    org.apache.spark.broadcast.TorrentBroadcast.<init> (TorrentBroadcast.scala:85)
    org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast (TorrentBroadcastFactory.scala:34)
    org.apache.spark.broadcast.BroadcastManager.newBroadcast (BroadcastManager.scala:63)
    org.apache.spark.SparkContext.broadcast (SparkContext.scala:1327)
    org.apache.spark.api.java.JavaSparkContext.broadcast (JavaSparkContext.scala:648)
    powderkeg.core$barrier_BANG_$fn__11172.invoke (core.clj:131)

Workaround:

Updating the version of kryo to 4.0.0 fixes this issue.

cgrand commented 7 years ago

This was similar to #11: in both cases I backported a Kryo fix.