BUGFIX: shutil.move can lead to "operation not permitted" error

I am using GEOparse on a platform with distributed docker images (FASTGenomics), where the user does not have root permissions. The system is Ubuntu 20.04 LTS. When I execute

gse = GEOparse.get_GEO("GSE6207", destdir="./GEO/")

I get the following error message:

10-Nov-2020 09:37:57 INFO GEOparse - Downloading ftp://ftp.ncbi.nlm.nih.gov/geo/series/GSE6nnn/GSE6207/soft/GSE6207_family.soft.gz to ./GEO/GSE6207_family.soft.gz
100%|██████████| 17.3M/17.3M [00:01<00:00, 9.59MB/s]
10-Nov-2020 09:38:00 DEBUG downloader - Size validation passed
10-Nov-2020 09:38:00 DEBUG downloader - Moving /tmp/tmpvhh3omno to /analysis/GEO/GSE6207_family.soft.gz
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
/opt/conda/lib/python3.8/shutil.py in move(src, dst, copy_function)
    787     try:
--> 788         os.rename(src, real_dst)
    789     except OSError:

OSError: [Errno 18] Invalid cross-device link: '/tmp/tmpvhh3omno' -> '/analysis/GEO/GSE6207_family.soft.gz'

During handling of the above exception, another exception occurred:

PermissionError                           Traceback (most recent call last)
/opt/conda/lib/python3.8/site-packages/GEOparse/utils.py in download_from_url(url, destination_path, force, aspera, silent)
     79         else:
---> 80             fn.download(silent=silent, force=force)
     81     except Exception as err:

/opt/conda/lib/python3.8/site-packages/GEOparse/downloader.py in download(self, force, silent)
     81             else:
---> 82                 _download()
     83         finally:

/opt/conda/lib/python3.8/site-packages/GEOparse/downloader.py in _download()
     56             logger.debug("Moving %s to %s" % (self._temp_file_name, self.destination))
---> 57             shutil.move(self._temp_file_name, self.destination)
     58             logger.debug("Successfully downloaded %s" % self.url)

/opt/conda/lib/python3.8/shutil.py in move(src, dst, copy_function)
    801         else:
--> 802             copy_function(src, real_dst)
    803             os.unlink(src)

/opt/conda/lib/python3.8/shutil.py in copy2(src, dst, follow_symlinks)
    432     copyfile(src, dst, follow_symlinks=follow_symlinks)
--> 433     copystat(src, dst, follow_symlinks=follow_symlinks)
    434     return dst

/opt/conda/lib/python3.8/shutil.py in copystat(src, dst, follow_symlinks)
    371     mode = stat.S_IMODE(st.st_mode)
--> 372     lookup("utime")(dst, ns=(st.st_atime_ns, st.st_mtime_ns),
    373         follow_symlinks=follow)

PermissionError: [Errno 1] Operation not permitted

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
<ipython-input-6-d172bf3b10bf> in <module>
----> 1 gse = GEOparse.get_GEO("GSE6207", destdir="./GEO/")

/opt/conda/lib/python3.8/site-packages/GEOparse/GEOparse.py in get_GEO(geo, filepath, destdir, how, annotate_gpl, geotype, include_data, silent, aspera, partial, open_kwargs)
     91 
     92     if filepath is None:
---> 93         filepath, geotype = get_GEO_file(
     94             geo,
     95             destdir=destdir,

/opt/conda/lib/python3.8/site-packages/GEOparse/GEOparse.py in get_GEO_file(geo, destdir, annotate_gpl, how, include_data, silent, aspera)
    250     if not path.isfile(filepath):
    251         logger.info("Downloading %s to %s" % (url, filepath))
--> 252         utils.download_from_url(url, filepath, silent=silent, aspera=aspera)
    253     else:
    254         logger.info("File already exist: using local version.")

/opt/conda/lib/python3.8/site-packages/GEOparse/utils.py in download_from_url(url, destination_path, force, aspera, silent)
     80             fn.download(silent=silent, force=force)
     81     except Exception as err:
---> 82         raise IOError(
     83             "Download failed due to '%s'. ID could be incorrect or the " % err
     84             + "data might not be public yet."

OSError: Download failed due to '[Errno 1] Operation not permitted'. ID could be incorrect or the data might not be public yet.

afaik the root of the problem is that shutil.move tries to set the owner/group + permission with chmod but the user has no permission to change these settings in my case. As there is anyway a os.remove(self._temp_file_name) command at the end, I would suggest to use shutil.copyfile instead of shutil.move.

guma44 / GEOparse

BUGFIX: shutil.move can lead to "operation not permitted" error #64