search

twitter / scalding

A Scala API for Cascading
http://twitter.com/scalding
Apache License 2.0
3.5k stars 706 forks source link

Introduce TypedPipe.toIterableExecutionWithClose and TypedPipe.useAsI… #1876

Closed dieu closed 5 years ago

dieu commented 5 years ago

Port of #1874 to 0.18.x

dieu commented 5 years ago

@johnynek I played more to make this PR workable and hit more problems with caching of ToWrite and then find out a new vulnerability with caching force. So in the end, looks like this feature is more complicated than needed, and option scalding.execution.cleanup.onfinish is good enough to achieve our goal with cleaning resources for long-running usage.

at this point, I think we should close this feature and not increase complexity. what's do you think?

johnynek commented 5 years ago

caching is hard...

I tend to agree with you.