search

twitter / scalding

A Scala API for Cascading
http://twitter.com/scalding
Apache License 2.0
3.49k stars 703 forks source link

Another example of an unplannable graph #1801

Open johnynek opened 6 years ago

johnynek commented 6 years ago 
- all optimization rules do not increase steps 

arg0 = WithDescriptionTypedPipe(Mapped(WithDescriptionTypedPipe(CrossPipe(WithDescriptionTypedPipe(Filter(IterablePipe(List(1406023175)),<function1>),List((org.scalacheck.Gen$R$class.map(Gen.scala:237),true))),WithDescriptionTypedPipe(TrappedPipe(WithDescriptionTypedPipe(ForceToDisk(WithDescriptionTypedPipe(Mapped(WithDescriptionTypedPipe(FilterKeys(WithDescriptionTypedPipe(Mapped(WithDescriptionTypedPipe(ForceToDisk(WithDescriptionTypedPipe(MergedTypedPipe(IterablePipe(List(1)),WithDescriptionTypedPipe(Fork(IterablePipe(List(1))),List((org.scalacheck.Gen$R$class.map(Gen.scala:237),true)))),List((org.scalacheck.Gen$R$class.map(Gen.scala:237),true)))),List((org.scalacheck.Gen$R$class.map(Gen.scala:237),true))),<function1>),List((org.scalacheck.Gen$R$class.map(Gen.scala:237),true))),<function1>),List((org.scalacheck.Gen$R$class.map(Gen.scala:237),true))),<function1>),List((org.scalacheck.Gen$R$class.map(Gen.scala:237),true)))),List((org.scalacheck.Gen$R$class.map(Gen.scala:237),true))),com.twitter.scalding.source.FixedTypedText(frco8uwemnb3cyHuwd9Feqeqrsc6ceqhsEnNOUmhbnk1apqnjs2IhzxMcg),Single(com.twitter.scalding.TupleGetter$IntGetter$@20d3d9e3)),List((org.scalacheck.Gen$R$class.map(Gen.scala:237),true)))),List((org.scalacheck.Gen$R$class.map(Gen.scala:237),true))),<function1>),List((org.scalacheck.Gen$R$class.map(Gen.scala:237),true))),
[info]       arg1 = com.twitter.scalding.typed.OptimizationRules$ComposeFlatMap$@6091c00e.orElse(com.twitter.scalding.typed.OptimizationRules$ComposeMap$@66a991c1).orElse(com.twitter.scalding.typed.OptimizationRules$ComposeFilter$@6c399402).orElse(com.twitter.scalding.typed.OptimizationRules$ComposeWithOnComplete$@3f0e4827).orElse(com.twitter.scalding.typed.OptimizationRules$ComposeMapFlatMap$@51717e78).orElse(com.twitter.scalding.typed.OptimizationRules$ComposeFilterFlatMap$@7b7cca0e).orElse(com.twitter.scalding.typed.OptimizationRules$EmptyIterableIsEmpty$@5bc5d8e7).orElse(com.twitter.scalding.typed.OptimizationRules$DescribeLater$@1647ecc8).orElse(com.twitter.scalding.typed.OptimizationRules$DiamondToFlatMap$@4c1fc2c6).orElse(com.twitter.scalding.typed.OptimizationRules$RemoveDuplicateForceFork$@51ae5df1).orElse(com.twitter.scalding.typed.OptimizationRules$IgnoreNoOpGroup$@24d9a5c3).orElse(com.twitter.scalding.typed.OptimizationRules$DeferMerge$@d513c3c).orElse(com.twitter.scalding.typed.OptimizationRules$FilterKeysEarly$@3feac6a2).orElse(com.twitter.scalding.typed.OptimizationRules$FilterLocally$@1c08457b).orElse(com.twitter.scalding.typed.OptimizationRules$EmptyIsOftenNoOp$@75960c84)

giving the cascading stack:

[info]   Cause: java.lang.NullPointerException:
[info]   at java.util.Objects.requireNonNull(Objects.java:203)
[info]   at java.util.Arrays$ArrayList.<init>(Arrays.java:3813)
[info]   at java.util.Arrays.asList(Arrays.java:3800)
[info]   at cascading.pipe.Splice.<init>(Splice.java:280)
[info]   at cascading.pipe.Splice.<init>(Splice.java:232)
[info]   at cascading.pipe.Splice.<init>(Splice.java:192)
[info]   at cascading.pipe.Splice.<init>(Splice.java:181)
[info]   at cascading.pipe.Splice.<init>(Splice.java:428)
[info]   at cascading.pipe.Merge.<init>(Merge.java:49)

I don't think this is a problem in production because we are testing unoptimized vs optimized graphs. The unoptimized graphs sometimes fail because cascading either has bugs or some other kinds of rules we don't follow about how Merge works. It would be nice to add a test for this graph so we can make sure even unoptimized scalding graphs will plan correctly.