twitter / scalding

A Scala API for Cascading
http://twitter.com/scalding
Apache License 2.0
3.49k stars 703 forks source link

add a config option to immediately clean up execution temp files #1808

Closed johnynek closed 6 years ago

johnynek commented 6 years ago

We register in execution a thread to run on shutdown to clean up.

This doesn't always (ever?) actually work because shutdown hooks seem unreliable here.

Alternatively, in many cases we know that when the execution is done, any ForceToDisk or ToIterable have been consumed, so we can clean up immediately. If we had a config option for this, we could be safer, and most users could turn this on by default in their ExecutionApp subclass that they use, or by passing in a config option.