twitter / scalding

A Scala API for Cascading
http://twitter.com/scalding
Apache License 2.0
3.49k stars 703 forks source link

Nested mutation bug #1819

Closed fwbrasil closed 6 years ago

fwbrasil commented 6 years ago

I'm debugging a test that fails on the latest develop revision. FlowStateMap.mutate calls are nested and mutate the same FlowDef. The outer mutation overrides the mutations done by the nested call. Stack trace from the debugging session:

screen shot 2018-02-20 at 1 37 37 pm

Note the two mutate calls.

Isolated example:

object NestedMutationBug extends App {
  implicit val fd = new FlowDef
  implicit val m = Local(false)
  val t = TypedPipe.from(TypedText.tsv[String]("")).write(NullSink)
  CascadingBackend.planTypedWrites(fd, m)
  val state = FlowStateMap.get(fd).get
  require(state.sourceMap.nonEmpty) // fails
}
johnynek commented 6 years ago

working on this now.