search

suzaku-io / boopickle

Binary serialization library for efficient network communication
Apache License 2.0
365 stars 41 forks source link

NPE when trying to serialize trait values with generic values using shapeless #103

Open bbarker opened 6 years ago

bbarker commented 6 years ago

This occurs in both JS and the JVM; I packaged up a somewhat minimal example of this. Here is the problematic trait:

  sealed trait JobWrap {
    type J
    val job: J
    val ev: Job[J]
  }

JVM errror:

[info] boopickling various API data structures
[info] - should not throw an error (46 milliseconds)
[info] - should not throw an error when JobWrapped *** FAILED *** (4 milliseconds)
[info]   java.lang.NullPointerException:
[info]   at boopickle.PickleImpl$.apply(Default.scala:76)
[info]   at boopickle.PickleImpl$.intoBytes(Default.scala:81)
[info]   at org.xsede.jobrunner.model.MainSpec.$anonfun$new$3(MainSpec.scala:32)
[info]   at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[info]   at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:22)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:20)
[info]   at org.scalatest.FlatSpecLike$$anon$1.apply(FlatSpecLike.scala:1691)
[info]   at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
[info]   at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
[info]   at org.scalatest.FlatSpec.withFixture(FlatSpec.scala:1685)
[info]   at org.scalatest.FlatSpecLike.invokeWithFixture$1(FlatSpecLike.scala:1689)
[info]   at org.scalatest.FlatSpecLike.$anonfun$runTest$1(FlatSpecLike.scala:1701)
[info]   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289)
[info]   at org.scalatest.FlatSpecLike.runTest(FlatSpecLike.scala:1701)
[info]   at org.scalatest.FlatSpecLike.runTest$(FlatSpecLike.scala:1683)
[info]   at org.scalatest.FlatSpec.runTest(FlatSpec.scala:1685)
[info]   at org.scalatest.FlatSpecLike.$anonfun$runTests$1(FlatSpecLike.scala:1759)
[info]   at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:396)
[info]   at scala.collection.immutable.List.foreach(List.scala:389)
[info]   at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
[info]   at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:373)
[info]   at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:410)
[info]   at scala.collection.immutable.List.foreach(List.scala:389)
[info]   at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
[info]   at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:379)
[info]   at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)
[info]   at org.scalatest.FlatSpecLike.runTests(FlatSpecLike.scala:1759)
[info]   at org.scalatest.FlatSpecLike.runTests$(FlatSpecLike.scala:1758)
[info]   at org.scalatest.FlatSpec.runTests(FlatSpec.scala:1685)
[info]   at org.scalatest.Suite.run(Suite.scala:1147)
[info]   at org.scalatest.Suite.run$(Suite.scala:1129)
[info]   at org.scalatest.FlatSpec.org$scalatest$FlatSpecLike$$super$run(FlatSpec.scala:1685)
[info]   at org.scalatest.FlatSpecLike.$anonfun$run$1(FlatSpecLike.scala:1804)
[info]   at org.scalatest.SuperEngine.runImpl(Engine.scala:521)
[info]   at org.scalatest.FlatSpecLike.run(FlatSpecLike.scala:1804)
[info]   at org.scalatest.FlatSpecLike.run$(FlatSpecLike.scala:1802)
[info]   at org.scalatest.FlatSpec.run(FlatSpec.scala:1685)
[info]   at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:318)
[info]   at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:513)
[info]   at sbt.TestRunner.runTest$1(TestFramework.scala:76)
[info]   at sbt.TestRunner.run(TestFramework.scala:85)
[info]   at sbt.TestFramework$$anon$2$$anonfun$$init$$1$$anonfun$apply$8.apply(TestFramework.scala:202)
[info]   at sbt.TestFramework$$anon$2$$anonfun$$init$$1$$anonfun$apply$8.apply(TestFramework.scala:202)
[info]   at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:185)
[info]   at sbt.TestFramework$$anon$2$$anonfun$$init$$1.apply(TestFramework.scala:202)
[info]   at sbt.TestFramework$$anon$2$$anonfun$$init$$1.apply(TestFramework.scala:202)
[info]   at sbt.TestFunction.apply(TestFramework.scala:207)
[info]   at sbt.Tests$$anonfun$9.apply(Tests.scala:216)
[info]   at sbt.Tests$$anonfun$9.apply(Tests.scala:216)
[info]   at sbt.std.Transform$$anon$3$$anonfun$apply$2.apply(System.scala:44)
[info]   at sbt.std.Transform$$anon$3$$anonfun$apply$2.apply(System.scala:44)
[info]   at sbt.std.Transform$$anon$4.work(System.scala:63)
[info]   at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
[info]   at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
[info]   at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
[info]   at sbt.Execute.work(Execute.scala:237)
[info]   at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
[info]   at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
[info]   at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
[info]   at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
[info]   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[info]   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[info]   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[info]   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[info]   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[info]   at java.lang.Thread.run(Thread.java:748)

JS error:

[info] boopickling various API data structures
[info] - should not throw an error (6 milliseconds)
[info] - should not throw an error when JobWrapped *** FAILED *** (2 milliseconds)
[info]   scala.scalajs.js.JavaScriptException: TypeError: Cannot read property 'pickle__O__Lboopickle_PickleState__V' of null
[info]   at boopickle.PickleImpl$.apply(/home/brandon/workspace/ProjectGists/Scala/BoopickleJobWrapCCRS/web-static/static/target/web-client-test-fastopt.js:2875:5)
[info]   at boopickle.PickleImpl$.intoBytes(/home/brandon/workspace/ProjectGists/Scala/BoopickleJobWrapCCRS/web-static/static/target/web-client-test-fastopt.js:2879:21)
[info]   at {anonymous}()(/home/brandon/workspace/ProjectGists/Scala/BoopickleJobWrapCCRS/web-static/static/target/web-client-test-fastopt.js:70295:80)
[info]   at scala.scalajs.runtime.AnonFunction0.apply(/home/brandon/workspace/ProjectGists/Scala/BoopickleJobWrapCCRS/web-static/static/target/web-client-test-fastopt.js:34267:23)
[info]   at org.scalatest.OutcomeOf.outcomeOf(/home/brandon/workspace/ProjectGists/Scala/BoopickleJobWrapCCRS/web-static/static/target/web-client-test-fastopt.js:1496:7)
[info]   at org.scalatest.Transformer.apply(/home/brandon/workspace/ProjectGists/Scala/BoopickleJobWrapCCRS/web-static/static/target/web-client-test-fastopt.js:46734:10)
[info]   at org.scalatest.FlatSpecLike$$anon$1.apply(/home/brandon/workspace/ProjectGists/Scala/BoopickleJobWrapCCRS/web-static/static/target/web-client-test-fastopt.js:35766:72)
[info]   at org.scalatest.FlatSpecLike.invokeWithFixture$1(/home/brandon/workspace/ProjectGists/Scala/BoopickleJobWrapCCRS/web-static/static/target/web-client-test-fastopt.js:65615:15)
[info]   at org.scalatest.FlatSpecLike.runTest(/home/brandon/workspace/ProjectGists/Scala/BoopickleJobWrapCCRS/web-static/static/target/web-client-test-fastopt.js:65818:5)
[info]   at {anonymous}()(/home/brandon/workspace/ProjectGists/Scala/BoopickleJobWrapCCRS/web-static/static/target/web-client-test-fastopt.js:65931:14)
ochrons commented 6 years ago

There is no way a macro could generate a pickler for such trait as it has no type parameters and no implementations, and could contain anything. Container types must have a way to reference the pickler for the contained data, You'll have to provide your own, custom pickler for such types.

cornerman commented 6 years ago

@bbarker What you are seeing here is one of the annoyances of scala's initializers, you have defined an implicit pickler in the jobwrap companion:

implicit val jwPickler: Pickler[JobWrap] = generatePickler[JobWrap]

Now, in the shapeless module, generatePickler is defined like this:

def generatePickler[T](implicit pickler: Pickler[T]): Pickler[T] = pickler

So, when calling generatePickler on the rhs of the implicit val jwPickler, it will resolve the implicit pickler with the currently uninitialized field jwPickler. In this case, the pickler is null. If you just try to call generatePickler without the implicit in scope, it will tell you that there is no derivable pickler.

In the normal boopickle.Default object, we defined the generatePickler method in a different way. It calls a macro directly. With shapeless, we get that implicitly.

@ochrons It think, we should make both functions behave the same way in order to avoid this unexpected behavior. But not sure yet, how we could do this here. What do you think?