twitter / scalding

A Scala API for Cascading
http://twitter.com/scalding
Apache License 2.0
3.5k stars 706 forks source link

Add unit tests for kryo serialization of toIterableExecution #1833

Closed tchow-stripe closed 6 years ago

tchow-stripe commented 6 years ago

an older version of kryo (2.21) fails to serialize results of toIterableExecution. Issue: https://github.com/EsotericSoftware/kryo/issues/15

Caused by: com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Can not set final com.twitter.scalding.typed.TypedPipe field com.twitter.scalding.Execution$$anonfun$toIterable$1.t$8 to com.twitter.scalding.Execution$ToWrite$ToIterable
Serialization trace:
t$8 (com.twitter.scalding.Execution$$anonfun$toIterable$1)
result (com.twitter.scalding.Execution$WriteExecution)
$outer (com.twitter.scalding.Execution$WriteExecution$$anonfun$map$1)
result (com.twitter.scalding.Execution$WriteExecution)
prev (com.twitter.scalding.Execution$FlatMapped)
usedSketch (com.stripe.zoolander.util.features.PrecomputedSketched$SketchInputs)
si$1 (com.stripe.zoolander.util.features.PrecomputedSketched$SketchedPipe$$anonfun$1)
fn$1 (com.stripe.zoolander.util.features.PrecomputedSketched$SketchedPipe$$anonfun$flatMapWithReplicas$1$1)
$outer (com.stripe.zoolander.util.features.PrecomputedSketched$SketchedPipe$$anonfun$flatMapWithReplicas$1$1$$anonfun$apply$2)
next (com.twitter.scalding.typed.functions.FlatMapFunctions$FromMapCompose)
next (com.twitter.scalding.typed.functions.FlatMapFunctions$FromFlatMapCompose)
com$twitter$scalding$typed$functions$FlatMappedFn$$toFn (com.twitter.scalding.typed.functions.FlatMappedFn$Series)
    at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:626)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:221)
    at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:648)
CLAassistant commented 6 years ago

CLA assistant check
Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.