Deal with inner case classes

[johnynek]

inner case classes seem to capture references to outer even if they are not used.

If we can find some cheap way to detect this, it would really help scalding and possibly spark.

[johnynek]

We could make a better version of the FieldsSerializer that works for case classes. These could be create at the first time the class is seen and could do that expensive check on outer being used the first time.

[clementgarnier]

Just ran into this issue – in my case it was throwing a java.util.ConcurrentModificationException, probably because something was modified in the outer class.

[johnynek]

We can't fix this example. That case class keeps a reference to test which is the outer class. That's just the way scala compiles it. We could try to determine that it is not needed by looking at all the bytecode and nulling it out in that case, but that would be a fairly non-trivial to do, and no one has yet taken it on.

Having people make the outer class serializable or removing the nesting is what we recommend. On Tue, Aug 1, 2017 at 07:54 Bekir Oguz notifications@github.com wrote:

I see this issue is open for a very long time already, I also have the same problem with FieldSerializer serializing case classes. The following little unit test can reproduce this.

import com.twitter.chill.{Input, Output, ScalaKryoInstantiator}import org.scalatest.{FunSuite, Matchers}

class KryoSerializerSpec extends FunSuite with Matchers {

case class Test1(a: Int, b: String)

test("kryo can serialize/deserialize case classes") { val buffer = new ArrayByte val input = new Input(buffer) val output = new Output(buffer) val obj = Test1(5, "test1")

val kryo = (new ScalaKryoInstantiator).newKryo()
kryo.writeObject(output, obj)
kryo.readObject(input, classOf[Test1]) shouldBe obj

}

}

This is the tracktrace:

java.util.ConcurrentModificationException Serialization trace: classes (sun.misc.Launcher$AppClassLoader) contextClassLoader (java.lang.Thread) value (java.util.concurrent.atomic.AtomicReference) atomic (org.scalatest.ConcurrentAlerter) value (java.util.concurrent.atomic.AtomicReference) atomicAlerter (org.scalatest.Engine) org$scalatest$FunSuiteLike$$engine (com.ing.baker.runtime.actor.KryoSerializerSpec) $outer (com.ing.baker.runtime.actor.KryoSerializerSpec$Test1) com.esotericsoftware.kryo.KryoException: java.util.ConcurrentModificationException Serialization trace: classes (sun.misc.Launcher$AppClassLoader) contextClassLoader (java.lang.Thread) value (java.util.concurrent.atomic.AtomicReference) atomic (org.scalatest.ConcurrentAlerter) value (java.util.concurrent.atomic.AtomicReference) atomicAlerter (org.scalatest.Engine) org$scalatest$FunSuiteLike$$engine (com.ing.baker.runtime.actor.KryoSerializerSpec) $outer (com.ing.baker.runtime.actor.KryoSerializerSpec$Test1) at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:101) at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:505) at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575) at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:80) at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:505) at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575) at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:80) at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:505) at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575) at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:80) at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:505) at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575) at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:80) at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:505) at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575) at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:80) at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:505) at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575) at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:80) at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:505) at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575) at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:80) at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:505) at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:557) at com.ing.baker.runtime.actor.KryoSerializerSpec$$anonfun$1.apply(KryoSerializerSpec.scala:18) at com.ing.baker.runtime.actor.KryoSerializerSpec$$anonfun$1.apply(KryoSerializerSpec.scala:11) at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85) at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) at org.scalatest.Transformer.apply(Transformer.scala:22) at org.scalatest.Transformer.apply(Transformer.scala:20) at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186) at org.scalatest.TestSuite$class.withFixture(TestSuite.scala:196) at org.scalatest.FunSuite.withFixture(FunSuite.scala:1560) at org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:183) at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:196) at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:196) at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289) at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:196) at org.scalatest.FunSuite.runTest(FunSuite.scala:1560) at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:229) at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:229) at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:396) at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:384) at scala.collection.immutable.List.foreach(List.scala:392) at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384) at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:379) at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461) at org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:229) at org.scalatest.FunSuite.runTests(FunSuite.scala:1560) at org.scalatest.Suite$class.run(Suite.scala:1147) at org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1560) at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:233) at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:233) at org.scalatest.SuperEngine.runImpl(Engine.scala:521) at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:233) at org.scalatest.FunSuite.run(FunSuite.scala:1560) at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:45) at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$1.apply(Runner.scala:1340) at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$1.apply(Runner.scala:1334) at scala.collection.immutable.List.foreach(List.scala:392) at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1334) at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1011) at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1010) at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1500) at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:1010) at org.scalatest.tools.Runner$.run(Runner.scala:850) at org.scalatest.tools.Runner.run(Runner.scala) at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runScalaTest2(ScalaTestRunner.java:138) at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:28) Caused by: java.util.ConcurrentModificationException at java.util.Vector$Itr.checkForComodification(Vector.java:1184) at java.util.Vector$Itr.next(Vector.java:1137) at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:99) at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40) at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575) at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:80) ... 68 more

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/twitter/chill/issues/122#issuecomment-319348960, or mute the thread https://github.com/notifications/unsubscribe-auth/AAEJdq0kz1LF8_JGSB0unnnIbIlHyz9Iks5sTxHVgaJpZM4A6UQm .