search

twitter / scalding

A Scala API for Cascading
http://twitter.com/scalding
Apache License 2.0
3.5k stars 706 forks source link

Potential planner bug in develop branch #1786

Closed johnynek closed 6 years ago

johnynek commented 6 years ago 
[info] - all optimization rules don't change results *** FAILED ***
[info]   NullPointerException was thrown during property evaluation.
[info]     Message: "None"
[info]     Occurred when passed generated values (
[info]       arg0 = WithDescriptionTypedPipe(TrappedPipe(WithDescriptionTypedPipe(MergedTypedPipe(WithDescriptionTypedPipe(EmptyTypedPipe,lVscjjUwdswuhslob3wcryitdoprmphqbl,false),WithDescriptionTypedPipe(EmptyTypedPipe,p8buqblzwenawcsgy8ckij7rndez4xlpcyJPwpzicbj9ct9hlekkzMhro5ugym9keiikxrjs7sutjuOhk,false)),org.scalacheck.Gen$R.map(Gen.scala:237),true),com.twitter.scalding.source.FixedTypedText(qayWlj3wtbwSlnkdhrgUTwilmjusWwsOdv7qdreth1933gtnuqzeythlyl6mbktjoimervxhhoavLbcyxB4duzrjxmk8b),Single(com.twitter.scalding.TupleGetter$IntGetter$@78d30dcc)),org.scalacheck.Gen$R.map(Gen.scala:237),true),
[info]       arg1 = com.twitter.scalding.typed.OptimizationRules$FilterLocally$@342d6071.orElse(com.twitter.scalding.typed.OptimizationRules$ComposeFlatMap$@12db1063).orElse(com.twitter.scalding.typed.OptimizationRules$ComposeMap$@41a47a5c).orElse(com.twitter.scalding.typed.OptimizationRules$ComposeFilter$@52cbaab4).orElse(com.twitter.scalding.typed.OptimizationRules$ComposeWithOnComplete$@d3a7674).orElse(com.twitter.scalding.typed.OptimizationRules$RemoveDuplicateForceFork$@47a827b8).orElse(com.twitter.scalding.typed.OptimizationRules$ForceToDiskBeforeHashJoin$@253cd38b).orElse(com.twitter.scalding.typed.OptimizationRules$DeferMerge$@2a2d4392).orElse(com.twitter.scalding.typed.OptimizationRules$EmptyIterableIsEmpty$@747a0975).orElse(com.twitter.scalding.typed.OptimizationRules$HashToShuffleCoGroup$@726975a5)
[info]     )

The stack that caused the error:

[info]   Cause: java.lang.NullPointerException:
[info]   at java.util.Objects.requireNonNull(Objects.java:203)
[info]   at java.util.Arrays$ArrayList.<init>(Arrays.java:3813)
[info]   at java.util.Arrays.asList(Arrays.java:3800)
[info]   at cascading.pipe.Splice.<init>(Splice.java:280)
[info]   at cascading.pipe.Splice.<init>(Splice.java:232)
[info]   at cascading.pipe.Splice.<init>(Splice.java:192)
[info]   at cascading.pipe.Splice.<init>(Splice.java:181)
[info]   at cascading.pipe.Splice.<init>(Splice.java:428)
[info]   at cascading.pipe.Merge.<init>(Merge.java:49)
[info]   at com.twitter.scalding.typed.cascading_backend.CascadingBackend$$anon$3.$anonfun$toFunction$1(CascadingBackend.scala:268)

from commit: https://github.com/twitter/scalding/pull/1784/commits/fa6eb87d80b996326455d6dfcf3bf3ef873190b4

oscar-stripe commented 6 years ago

Here's what looks like another hit of this:

[info] - all optimization rules don't change results *** FAILED ***
[info]   NullPointerException was thrown during property evaluation.
[info]     Message: "None"
[info]     Occurred when passed generated values (
[info]       arg0 = WithDescriptionTypedPipe(Mapped(WithDescriptionTypedPipe(CrossPipe(WithDescriptionTypedPipe(MergedTypedPipe(IterablePipe(List(2147483647)),IterablePipe(List(2147483647))),org.scalacheck.Gen$R.map(Gen.scala:237),true),IterablePipe(List(-1))),org.scalacheck.Gen$R.map(Gen.scala:237),true),<function1>),org.scalacheck.Gen$R.map(Gen.scala:237),true),
[info]       arg1 = com.twitter.scalding.typed.OptimizationRules$AddExplicitForks$@1bc0b0e2.orElse(com.twitter.scalding.typed.OptimizationRules$ComposeFlatMap$@47811af2).orElse(com.twitter.scalding.typed.OptimizationRules$EmptyIterableIsEmpty$@48f48b19).orElse(com.twitter.scalding.typed.OptimizationRules$ForceToDiskBeforeHashJoin$@75940cb2).orElse(com.twitter.scalding.typed.OptimizationRules$EmptyIsOftenNoOp$@6bcde542).orElse(com.twitter.scalding.typed.OptimizationRules$ComposeMapFlatMap$@24543443).orElse(com.twitter.scalding.typed.OptimizationRules$FilterKeysEarly$@4894cf93).orElse(com.twitter.scalding.typed.OptimizationRules$HashToShuffleCoGroup$@1db0bbaf).orElse(com.twitter.scalding.typed.OptimizationRules$DescribeLater$@1953aaae)
[info]     )

which has come up on another branch

Will add a test for that one too.