templateflow / python-client

A python client to query TemplateFlow via pyBIDS
https://templateflow.org/python-client/
Apache License 2.0
8 stars 12 forks source link

Autoupdate with overwrite ``True`` does not really overwrite. #62

Closed oesteban closed 6 months ago

oesteban commented 3 years ago

Identified in dMRIPrep: https://travis-ci.org/github/nipreps/dmriprep/jobs/735673066

___________________ ERROR collecting dmriprep/cli/parser.py ____________________
dmriprep/cli/parser.py:6: in <module>
    from .. import config
dmriprep/config/__init__.py:99: in <module>
    from niworkflows.utils.spaces import SpatialReferences as _SRs, Reference as _Ref
/tmp/venv/lib/python3.6/site-packages/niworkflows/utils/spaces.py:6: in <module>
    from templateflow import api as _tfapi
/tmp/venv/lib/python3.6/site-packages/templateflow/__init__.py:19: in <module>
    from . import api
/tmp/venv/lib/python3.6/site-packages/templateflow/api.py:7: in <module>
    from .conf import TF_LAYOUT, TF_S3_ROOT, TF_USE_DATALAD
/tmp/venv/lib/python3.6/site-packages/templateflow/conf/__init__.py:42: in <module>
    _update_s3(TF_HOME, local=True, overwrite=True)
/tmp/venv/lib/python3.6/site-packages/templateflow/conf/_s3.py:20: in update
    retval = _update_skeleton(skel_file, dest, overwrite=overwrite, silent=silent)
/tmp/venv/lib/python3.6/site-packages/templateflow/conf/_s3.py:55: in _update_skeleton
    zipref.extractall(str(dest))
/opt/python/3.6.7/lib/python3.6/zipfile.py:1524: in extractall
    self._extract_member(zipinfo, path, pwd)
/opt/python/3.6.7/lib/python3.6/zipfile.py:1574: in _extract_member
    os.mkdir(targetpath)
E   FileExistsError: [Errno 17] File exists: '/home/travis/.cache/templateflow/tpl-MNI152Lin'
oesteban commented 6 months ago

As @effigies noted in the duplicate:

I think the fix here is that, when we encounter a FileExistsError in _update_s3(), we should check that the files that were intended to be extracted match checksums and succeed if they do rather than erroring.

Might be a better place for the try/except, though.