psychoinformatics-de / fairly-big-processing-workflow

A template for decentralized, reproducible processing
Other
14 stars 3 forks source link

What to do when HPC doesn't delete files #2

Open mattcieslak opened 3 years ago

mattcieslak commented 3 years ago

Hi! We recently discovered that our HPC doesn't automatically clean up scratch files when a job finishes. Could you suggest a few commands that efficiently delete the temporary copy of the dataset?

When we work with temporary clones we typically get some data, interact with it, then git annex dead here. After that we'll move to the parent directory, rm -rf ds, then get permissions errors on some of the files, then chmod -R +w ds and finally rm -rf ds again. Is there a better way to do this?