Currently, Jobson will leave the working directories intact after a job has finished. This is useful for debugging the system. However, some of the jobs we run internally can produce >20 GB of data, which might still sit around after Jobson copies it to the final output location.
I usually use a separate cronjob to delete older working directories. However, if there's a lot of requests in a short period of time the disk will just fill up before the cronjobs get a chance to clean everything up.
For this fix I will:
Add a cleanup_working_directories: [false] property into the config. Default behavior will be to not cleanup the directories, in order to maintain backwards compat.
Make the job pipeline use that config property to cleanup working dirs
Currently, Jobson will leave the working directories intact after a job has finished. This is useful for debugging the system. However, some of the jobs we run internally can produce >20 GB of data, which might still sit around after Jobson copies it to the final output location.
I usually use a separate cronjob to delete older working directories. However, if there's a lot of requests in a short period of time the disk will just fill up before the cronjobs get a chance to clean everything up.
For this fix I will:
cleanup_working_directories: [false]
property into the config. Default behavior will be to not cleanup the directories, in order to maintain backwards compat.