twitter / scalding

A Scala API for Cascading
http://twitter.com/scalding
Apache License 2.0
3.48k stars 704 forks source link

Check if flow doesn't have sinks after we build it #1880

Closed dieu closed 5 years ago

dieu commented 5 years ago

Hey,

When we work with FlowDef and Execution.fromFn, writes not inserted directly into flowDef anymore, so we need to buildFlow first and then check if the flow doesn't have sinks. Otherwise, we will always have empty flowDef.

dieu commented 5 years ago

@johnynek added the test for check use case, but unfortunately, I have to change the signature of buildFlow from Try[Flow[_]] to Try[Option[Flow[_]]] because we can have totally empty flowDef in case of toIterableExcution when we don't need to do anything only addIter. what do you think?

We can use special exception and keep the signature as is, but I'm not sure that's a good approach.