search

twitter / scalding

A Scala API for Cascading
http://twitter.com/scalding
Apache License 2.0
3.5k stars 706 forks source link

Pull cascading flowDef thread into cascading_backend #1681

Closed johnynek closed 7 years ago

johnynek commented 7 years ago

This is a first step towards factoring Execution so that it does not know about the backend. This is a good cleanup regardless, because we separate the concern of caching from producing Futures from FlowDefs.

But I want to drive towards the Mode providing something to take a sequence of TypedPipe[T], TypedSink[T] pairs and returning a future Unit (and maybe counters) when they have all been written.

ianoc commented 7 years ago

👍 lgtm

johnynek commented 7 years ago

merging now, and can continue fixes in the next in the series.