search

twitter / scalding

A Scala API for Cascading
http://twitter.com/scalding
Apache License 2.0
3.5k stars 706 forks source link

Move toIterableExecution and forceToDiskExecution into Execution #1682

Closed johnynek closed 7 years ago

johnynek commented 7 years ago

This is a follow up to #1681

Here we have done almost all of the work to make Execution delegate to a Writer that handles all the side effects.

With this we can basically do 100% of having an in-memory or spark backend as long as users don't use certain cascading only features (toPipe, TypedPipe from a Pipe, or Execution from a FlowDef).

We are getting very close to where I want to be for the next published version. I want to make an example in-memory backend just to exercise everything to see if we really can run non-trivial jobs. We can leave a spark or flink backend to a future work.

I would like to remove the Execution from a FlowDef, I doubt it was ever used. I'd like to move toPipe to an implicit enrichment that users opt into if they really need it (which I doubt they would).

johnynek commented 7 years ago

@ianoc any time for this one?

johnynek commented 7 years ago

ping @benpence

ianoc commented 7 years ago

lgtm