The job rename_dset_storage_loc uses shutil.move for moving the dataset. This uses os.rename and if it doesnt work (e.g. renaming a non-leaf dir) it will use copy instead, whereafter it will remove the src dir.
Make this operation more atomic by using os.renames(src, dst), which automatically creates non-existing parent dirs. Test that it works.
The job
rename_dset_storage_locuses shutil.move for moving the dataset. This uses os.rename and if it doesnt work (e.g. renaming a non-leaf dir) it will use copy instead, whereafter it will remove the src dir. Make this operation more atomic by usingos.renames(src, dst), which automatically creates non-existing parent dirs. Test that it works.