Really weird bug that took me a while to figure out. But was seeing errors from running Snakemake: https://pastebin.com/uBmBwdnK. The snakemake pipeline still ran successfully, but threw this error as part of the clean up step.
It seems like snakemake doesn't properly close it's file descriptor when opening up the abc_env.yml file up in its cache. As a result, when it later tries to delete the file, the file gets renamed to '.nfs002...'. And a later step can't delete the directory because it's not empty.
The fix is to use an older version of smart_open, which handles closing the fd properly after it's used
Really weird bug that took me a while to figure out. But was seeing errors from running Snakemake: https://pastebin.com/uBmBwdnK. The snakemake pipeline still ran successfully, but threw this error as part of the clean up step.
It seems like snakemake doesn't properly close it's file descriptor when opening up the abc_env.yml file up in its cache. As a result, when it later tries to delete the file, the file gets renamed to '.nfs002...'. And a later step can't delete the directory because it's not empty.
The fix is to use an older version of smart_open, which handles closing the fd properly after it's used