search

twitter / scalding

A Scala API for Cascading
http://twitter.com/scalding
Apache License 2.0
3.49k stars 703 forks source link

Add an option to cleanup temp Execution files on finish #1810

Closed johnynek closed 6 years ago

johnynek commented 6 years ago

closes #1808

This adds a new option to remove temp files in cascading-based Executions, when the outer-most execution completes. This is safe to do when the outer-most execution is an Execution[Unit], which it usually is for an application. So in Job.toExecution or ExecutionApp, this can be the default setting. Otherwise, we default to false.

ianoc commented 6 years ago

modulo using a thread to actually do the cleanup -- lgtm

johnynek commented 6 years ago

Thank you very much for your time on the review!