adamkewley / jobson

A platform for transforming command-line applications into a job service.
Apache License 2.0
256 stars 20 forks source link

Add support for working directory cleanup #21

Closed adamkewley closed 6 years ago

adamkewley commented 6 years ago

Currently, Jobson will leave the working directories intact after a job has finished. This is useful for debugging the system. However, some of the jobs we run internally can produce >20 GB of data, which might still sit around after Jobson copies it to the final output location.

I usually use a separate cronjob to delete older working directories. However, if there's a lot of requests in a short period of time the disk will just fill up before the cronjobs get a chance to clean everything up.

For this fix I will:

adamkewley commented 6 years ago

This has now been implemented in the master branch