search

sxs-collaboration / sxs

Python code for manipulating data from the SXS collaboration
MIT License
25 stars 18 forks source link

tempfile replace problems #16

Closed moble closed 3 years ago

moble commented 3 years ago

A user sent me this:

I got the following error message when I was trying to load the catalog:

catalog = sxs.load("catalog") Traceback (most recent call last): File "", line 1, in File "/home/the_user_name/anaconda3/lib/python3.8/site-packages/sxs/handlers.py", line 213, in load return Catalog.load(download=download) File "/home/the_user_name/anaconda3/lib/python3.8/site-packages/sxs/catalog/catalog.py", line 82, in load zip_path.replace(cache_path) File "/home/the_user_name/anaconda3/lib/python3.8/pathlib.py", line 1369, in replace self._accessor.replace(self, target) OSError: [Errno 18] Invalid cross-device link: '/tmp/tmp9yb781zh/catalog.zip' -> '/home/the_user_name/.cache/sxs/catalog.zip'

It looks like one line (function .replace) in catalog.py is not compatible with all systems. You need to charge from .replace to shutil.move in order to fix this issue:

/anaconda3/lib/python3.8/site-packages/sxs/catalog/catalog.py

Line 4: Add:

  import shutil

Line 82: Change

zip_path.replace(cache_path)

dest_path = shutil.move(zip_path, cache_path)

On further investigation, I find that the replace function only works if the source and destination are on the same filesystem — suggesting that python's tempfile module made a temporary directory on some disk other than the one holding the user's home directory. This seems like it could be a fairly common use case.

I think a more robust fix would be to ask for the temporary files in the same folder as we expect to write the catalog zip file in, because read/write errors should occur if and only if there will be errors for the final file. Plus, there will be no copying between file systems.