Failing to clean /tmp when /tmp is mounted from a remote location.

[mjpdejong]

Describe the bug When updating from medaka 1.7.2 to medaka 1.7.3 I run into a problem concerning the usage of tmp. Medaka fails to clear the tmpdir after completion with the current medaka version.

Individual nodes on our HPC have a fairly small amount of tmp space and for this reason it is default for us to mount a fast remote infiniband share to use as tmp directory. It seems however that with the latest update the tmp cleanup runs into an error. If I switch to the local /tmp all is fine. While the /tmp usage in this small enough to fit in our /tmp we could work around this problem, but it requires an exception for medaka. Since I do not like making exceptions, and it is not really clear to me whether this is a feature or a bug, I made this report.

Greets, Marco

module load singularity/3.8.4

singularity exec --bind /mnt:/mnt --bind /data:/tmp /mnt/path_to_medaka/medaka_20230329.simg medaka consensus --threads 8 --model r941_sup_plant_g610 --batch_size 100 /mnt/path_to_bam/input.bam /mnt/path_to_output/output.hdf

Logging

[12:14:33 - Predict] Finished processing all regions.
Traceback (most recent call last):
  File "/usr/local/bin/medaka", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/dist-packages/medaka/medaka.py", line 724, in main
    args.func(args)
  File "/usr/local/lib/python3.8/dist-packages/medaka/prediction.py", line 205, in predict
    pass
  File "/usr/local/lib/python3.8/dist-packages/medaka/datastore.py", line 162, in __exit__
    self.cleanup()
  File "/usr/local/lib/python3.8/dist-packages/medaka/datastore.py", line 152, in cleanup
    self._exitstack.close()
  File "/usr/lib/python3.8/contextlib.py", line 533, in close
    self.__exit__(None, None, None)
  File "/usr/lib/python3.8/contextlib.py", line 525, in __exit__
    raise exc_details[1]
  File "/usr/lib/python3.8/contextlib.py", line 510, in __exit__
    if cb(*exc_details):
  File "/usr/lib/python3.8/tempfile.py", line 966, in __exit__
    self.cleanup()
  File "/usr/lib/python3.8/tempfile.py", line 970, in cleanup
    self._rmtree(self.name)
  File "/usr/lib/python3.8/tempfile.py", line 952, in _rmtree
    _rmtree(name, onerror=onerror)
  File "/usr/lib/python3.8/shutil.py", line 718, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/lib/python3.8/shutil.py", line 655, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib/python3.8/shutil.py", line 659, in _rmtree_safe_fd
    onerror(os.rmdir, fullname, sys.exc_info())
  File "/usr/lib/python3.8/shutil.py", line 657, in _rmtree_safe_fd
    os.rmdir(entry.name, dir_fd=topfd)
OSError: [Errno 39] Directory not empty: 'variables'

Environment (if you do not have a GPU, write No GPU): NO GPU pip install in docker (converted to singularity) - Ubuntu 20.04 medaka 1.7.3

[cjw85]

Hi @mjpdejong,

Some code was added in 1.7.3 to mitigate a potential security issue in the Python standard library CVE-2007-4559. I must admit, I don't understand how this change would lead to the error you are seeing. I would need refresh my memory of the code.

For whatever reason Python is having trouble recursively removing files that it has unpacked onto your filesystem at /data.

[cjw85]

Closing as duplicate.