Open mathieuboudreau opened 3 days ago
Had to revert back to repo2data-2.9.1 for this specific error to stop, but then I get,
mathieuboudreau@Mathieus-MacBook-Pro binder % repo2data -r data_requirement.json
---- repo2data starting ----
/Users/mathieuboudreau/opt/anaconda3/lib/python3.12/site-packages/repo2data
Config from file :
data_requirement.json
Destination:
../data/mono_t2
Info : Starting to download from osf https://osf.io/download/kujp3/ ...
Traceback (most recent call last):
File "/Users/mathieuboudreau/opt/anaconda3/bin/osf", line 8, in <module>
sys.exit(main())
^^^^^^
File "/Users/mathieuboudreau/opt/anaconda3/lib/python3.12/site-packages/osfclient/__main__.py", line 120, in main
exit_code = args.func(args)
^^^^^^^^^^^^^^^
File "/Users/mathieuboudreau/opt/anaconda3/lib/python3.12/site-packages/osfclient/cli.py", line 98, in wrapper
return_value = f(cli_args)
^^^^^^^^^^^
File "/Users/mathieuboudreau/opt/anaconda3/lib/python3.12/site-packages/osfclient/cli.py", line 158, in clone
project = osf.project(args.project)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/mathieuboudreau/opt/anaconda3/lib/python3.12/site-packages/osfclient/api.py", line 33, in project
type_ = self.guid(project_id)
^^^^^^^^^^^^^^^^^^^^^
File "/Users/mathieuboudreau/opt/anaconda3/lib/python3.12/site-packages/osfclient/api.py", line 41, in guid
return self._json(self._get(self._build_url('guids', guid)), 200)['data']['type']
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/mathieuboudreau/opt/anaconda3/lib/python3.12/site-packages/osfclient/models/core.py", line 61, in _json
raise RuntimeError("Response has status "
RuntimeError: Response has status code 404 not (200,)
Traceback (most recent call last):
File "/Users/mathieuboudreau/opt/anaconda3/bin/repo2data", line 60, in <module>
main()
File "/Users/mathieuboudreau/opt/anaconda3/bin/repo2data", line 57, in main
repo2data.install()
File "/Users/mathieuboudreau/opt/anaconda3/lib/python3.12/site-packages/repo2data/repo2data.py", line 102, in install
self._data_requirement_path,key).install()]
^^^^^^^^^
File "/Users/mathieuboudreau/opt/anaconda3/lib/python3.12/site-packages/repo2data/repo2data.py", line 360, in install
self._scan_dl_type()
File "/Users/mathieuboudreau/opt/anaconda3/lib/python3.12/site-packages/repo2data/repo2data.py", line 343, in _scan_dl_type
self._osf_download()
File "/Users/mathieuboudreau/opt/anaconda3/lib/python3.12/site-packages/repo2data/repo2data.py", line 303, in _osf_download
subprocess.check_call(
File "/Users/mathieuboudreau/opt/anaconda3/lib/python3.12/subprocess.py", line 413, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['osf', '--project', 'downl', 'clone', '../data/mono_t2']' returned non-zero exit status 1.
mathieuboudreau@Mathieus-MacBook-Pro binder %
That error may be related to be the same one I previously reported,
Trying a manual install of the repo through github, running the tests locally I get
mathieuboudreau@Mathieus-MacBook-Pro Repo2Data % pytest tests
============================================= test session starts =============================================
platform darwin -- Python 3.12.2, pytest-8.3.3, pluggy-1.5.0
rootdir: /Users/mathieuboudreau/neuropoly/mooc/repos/mooc-data/03-T2/Repo2Data
configfile: pyproject.toml
plugins: anyio-4.2.0
collected 10 items / 1 error
=================================================== ERRORS ====================================================
____________________________________ ERROR collecting tests/test_gdrive.py ____________________________________
/Users/mathieuboudreau/neuropoly/mooc/repos/mooc-data/03-T2/Repo2Data/tests/test_gdrive.py:30: in <module>
t.test_gdrive()
/Users/mathieuboudreau/neuropoly/mooc/repos/mooc-data/03-T2/Repo2Data/tests/test_gdrive.py:25: in test_gdrive
repo2data.install()
/Users/mathieuboudreau/neuropoly/mooc/repos/mooc-data/03-T2/Repo2Data/repo2data/repo2data.py:110: in install
self._use_server, self._data_requirement_path, None, self._server_dst_folder).install()]
/Users/mathieuboudreau/neuropoly/mooc/repos/mooc-data/03-T2/Repo2Data/repo2data/repo2data.py:364: in install
self._scan_dl_type()
/Users/mathieuboudreau/neuropoly/mooc/repos/mooc-data/03-T2/Repo2Data/repo2data/repo2data.py:335: in _scan_dl_type
self._gdrive_download()
/Users/mathieuboudreau/neuropoly/mooc/repos/mooc-data/03-T2/Repo2Data/repo2data/repo2data.py:251: in _gdrive_download
subprocess.check_call(
/Users/mathieuboudreau/opt/anaconda3/lib/python3.12/subprocess.py:413: in check_call
raise CalledProcessError(retcode, cmd)
E subprocess.CalledProcessError: Command '['gdown', 'https://drive.google.com/uc?id=1_zeJqQP8umrTk-evSAt3wCLxAkTKo0lC']' returned non-zero exit status 1.
----------------------------------------------- Captured stdout -----------------------------------------------
---- repo2data starting ----
/Users/mathieuboudreau/neuropoly/mooc/repos/mooc-data/03-T2/Repo2Data/repo2data
Config from file :
./tests/in/gdrive.json
Destination:
./tests/out/repo2data_gdrive
Info : Starting to download from Google drive https://drive.google.com/uc?id=1_zeJqQP8umrTk-evSAt3wCLxAkTKo0lC ...
Access denied with the following error:
----------------------------------------------- Captured stderr -----------------------------------------------
Cannot retrieve the public link of the file. You may need to change
the permission to 'Anyone with the link', or have had many accesses.
You may still be able to access the file from the browser:
https://drive.google.com/uc?id=1_zeJqQP8umrTk-evSAt3wCLxAkTKo0lC
============================================== warnings summary ===============================================
repo2data/repo2data.py:343
/Users/mathieuboudreau/neuropoly/mooc/repos/mooc-data/03-T2/Repo2Data/repo2data/repo2data.py:343: SyntaxWarning: invalid escape sequence '\.'
elif re.match(".*?(10\.\d{4}/zenodo).*?", self._data_requirement_file["src"]):
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================================== short test summary info ===========================================
ERROR tests/test_gdrive.py - subprocess.CalledProcessError: Command '['gdown', 'https://drive.google.com/uc?id=1_zeJqQP8umrTk-evSAt3wCL...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
========================================= 1 warning, 1 error in 1.56s =========================================
mathieuboudreau@Mathieus-MacBook-Pro Repo2Data %
Is there a CI running the tests prior to pushing the wheels to pipy? I don't see any in GitHub Actions, but see that there is also a CircleCI folder
Thanks @mathieuboudreau I'll look into this. I've changed release strategy, it must be having issues linking executables.
It's odd though, because wouldn't it be breaking all the other installs using this, i.e. through github actions? Pretty sure I've used some more recently than that push without issue; let me test elsewhere to make sure it's not a weird python env problem on my local
Ok so it's not just me, I just reproduced on a clean MyBinder session through the terminal:
jovyan@jupyter-qmrlab-2dmooc-2ddata-2dnczdjebs:~$ pip install repo2data
Collecting repo2data
Downloading repo2data-2.9.6-py3-none-any.whl.metadata (12 kB)
Collecting awscli (from repo2data)
Downloading awscli-1.34.31-py3-none-any.whl.metadata (11 kB)
Collecting patool (from repo2data)
Downloading patool-3.0.0-py2.py3-none-any.whl.metadata (4.0 kB)
Collecting datalad (from repo2data)
Downloading datalad-1.1.3-py3-none-any.whl.metadata (36 kB)
Requirement already satisfied: requests in /srv/conda/envs/notebook/lib/python3.10/site-packages (from repo2data) (2.32.3)
Collecting osfclient (from repo2data)
Downloading osfclient-0.0.5-py2.py3-none-any.whl.metadata (5.5 kB)
Collecting gdown (from repo2data)
Downloading gdown-5.2.0-py3-none-any.whl.metadata (5.8 kB)
Collecting zenodo-get (from repo2data)
Downloading zenodo_get-1.6.1-py2.py3-none-any.whl.metadata (1.1 kB)
Collecting botocore==1.35.31 (from awscli->repo2data)
Downloading botocore-1.35.31-py3-none-any.whl.metadata (5.6 kB)
Collecting docutils<0.17,>=0.10 (from awscli->repo2data)
Downloading docutils-0.16-py2.py3-none-any.whl.metadata (2.7 kB)
Collecting s3transfer<0.11.0,>=0.10.0 (from awscli->repo2data)
Downloading s3transfer-0.10.2-py3-none-any.whl.metadata (1.7 kB)
Requirement already satisfied: PyYAML<6.1,>=3.10 in /srv/conda/envs/notebook/lib/python3.10/site-packages (from awscli->repo2data) (6.0.2)
Collecting colorama<0.4.7,>=0.2.5 (from awscli->repo2data)
Downloading colorama-0.4.6-py2.py3-none-any.whl.metadata (17 kB)
Collecting rsa<4.8,>=3.1.2 (from awscli->repo2data)
Downloading rsa-4.7.2-py3-none-any.whl.metadata (3.6 kB)
Collecting jmespath<2.0.0,>=0.7.1 (from botocore==1.35.31->awscli->repo2data)
Downloading jmespath-1.0.1-py3-none-any.whl.metadata (7.6 kB)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /srv/conda/envs/notebook/lib/python3.10/site-packages (from botocore==1.35.31->awscli->repo2data) (2.9.0)
Requirement already satisfied: urllib3!=2.2.0,<3,>=1.25.4 in /srv/conda/envs/notebook/lib/python3.10/site-packages (from botocore==1.35.31->awscli->repo2data) (2.2.2)
Requirement already satisfied: platformdirs in /srv/conda/envs/notebook/lib/python3.10/site-packages (from datalad->repo2data) (4.2.2)
Collecting chardet>=3.0.4 (from datalad->repo2data)
Downloading chardet-5.2.0-py3-none-any.whl.metadata (3.4 kB)
Collecting iso8601 (from datalad->repo2data)
Downloading iso8601-2.1.0-py3-none-any.whl.metadata (3.7 kB)
Collecting humanize (from datalad->repo2data)
Downloading humanize-4.10.0-py3-none-any.whl.metadata (7.9 kB)
Collecting fasteners>=0.14 (from datalad->repo2data)
Downloading fasteners-0.19-py3-none-any.whl.metadata (4.9 kB)
Requirement already satisfied: packaging in /srv/conda/envs/notebook/lib/python3.10/site-packages (from datalad->repo2data) (24.1)
Collecting tqdm>=4.32.0 (from datalad->repo2data)
Downloading tqdm-4.66.5-py3-none-any.whl.metadata (57 kB)
Collecting annexremote (from datalad->repo2data)
Downloading annexremote-1.6.6-py3-none-any.whl.metadata (6.5 kB)
Collecting looseversion (from datalad->repo2data)
Downloading looseversion-1.3.0-py2.py3-none-any.whl.metadata (4.6 kB)
Collecting boto3 (from datalad->repo2data)
Downloading boto3-1.35.31-py3-none-any.whl.metadata (6.6 kB)
Collecting keyring!=23.9.0,>=20.0 (from datalad->repo2data)
Downloading keyring-25.4.1-py3-none-any.whl.metadata (20 kB)
Collecting keyrings.alt (from datalad->repo2data)
Downloading keyrings.alt-5.0.2-py3-none-any.whl.metadata (3.6 kB)
Collecting msgpack (from datalad->repo2data)
Downloading msgpack-1.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (8.4 kB)
Collecting python-gitlab (from datalad->repo2data)
Downloading python_gitlab-4.12.2-py3-none-any.whl.metadata (8.3 kB)
Requirement already satisfied: typing-extensions>=4.0.0 in /srv/conda/envs/notebook/lib/python3.10/site-packages (from datalad->repo2data) (4.12.2)
Collecting distro (from datalad->repo2data)
Downloading distro-1.9.0-py3-none-any.whl.metadata (6.8 kB)
Requirement already satisfied: charset-normalizer<4,>=2 in /srv/conda/envs/notebook/lib/python3.10/site-packages (from requests->repo2data) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /srv/conda/envs/notebook/lib/python3.10/site-packages (from requests->repo2data) (3.8)
Requirement already satisfied: certifi>=2017.4.17 in /srv/conda/envs/notebook/lib/python3.10/site-packages (from requests->repo2data) (2024.7.4)
Requirement already satisfied: beautifulsoup4 in /srv/conda/envs/notebook/lib/python3.10/site-packages (from gdown->repo2data) (4.12.3)
Collecting filelock (from gdown->repo2data)
Downloading filelock-3.16.1-py3-none-any.whl.metadata (2.9 kB)
Requirement already satisfied: six in /srv/conda/envs/notebook/lib/python3.10/site-packages (from osfclient->repo2data) (1.16.0)
Collecting wget (from zenodo-get->repo2data)
Downloading wget-3.2.zip (10 kB)
Preparing metadata (setup.py) ... done
Collecting jaraco.classes (from keyring!=23.9.0,>=20.0->datalad->repo2data)
Downloading jaraco.classes-3.4.0-py3-none-any.whl.metadata (2.6 kB)
Collecting jaraco.functools (from keyring!=23.9.0,>=20.0->datalad->repo2data)
Downloading jaraco.functools-4.1.0-py3-none-any.whl.metadata (2.9 kB)
Collecting jaraco.context (from keyring!=23.9.0,>=20.0->datalad->repo2data)
Downloading jaraco.context-6.0.1-py3-none-any.whl.metadata (4.1 kB)
Requirement already satisfied: importlib-metadata>=4.11.4 in /srv/conda/envs/notebook/lib/python3.10/site-packages (from keyring!=23.9.0,>=20.0->datalad->repo2data) (8.4.0)
Collecting SecretStorage>=3.2 (from keyring!=23.9.0,>=20.0->datalad->repo2data)
Downloading SecretStorage-3.3.3-py3-none-any.whl.metadata (4.0 kB)
Collecting jeepney>=0.4.2 (from keyring!=23.9.0,>=20.0->datalad->repo2data)
Downloading jeepney-0.8.0-py3-none-any.whl.metadata (1.3 kB)
Collecting pyasn1>=0.1.3 (from rsa<4.8,>=3.1.2->awscli->repo2data)
Downloading pyasn1-0.6.1-py3-none-any.whl.metadata (8.4 kB)
Requirement already satisfied: soupsieve>1.2 in /srv/conda/envs/notebook/lib/python3.10/site-packages (from beautifulsoup4->gdown->repo2data) (2.5)
Collecting requests-toolbelt>=1.0.0 (from python-gitlab->datalad->repo2data)
Downloading requests_toolbelt-1.0.0-py2.py3-none-any.whl.metadata (14 kB)
Requirement already satisfied: PySocks!=1.5.7,>=1.5.6 in /srv/conda/envs/notebook/lib/python3.10/site-packages (from requests[socks]->gdown->repo2data) (1.7.1)
Requirement already satisfied: zipp>=0.5 in /srv/conda/envs/notebook/lib/python3.10/site-packages (from importlib-metadata>=4.11.4->keyring!=23.9.0,>=20.0->datalad->repo2data) (3.20.1)
Requirement already satisfied: cryptography>=2.0 in /srv/conda/envs/notebook/lib/python3.10/site-packages (from SecretStorage>=3.2->keyring!=23.9.0,>=20.0->datalad->repo2data) (43.0.0)
Collecting more-itertools (from jaraco.classes->keyring!=23.9.0,>=20.0->datalad->repo2data)
Downloading more_itertools-10.5.0-py3-none-any.whl.metadata (36 kB)
Collecting backports.tarfile (from jaraco.context->keyring!=23.9.0,>=20.0->datalad->repo2data)
Downloading backports.tarfile-1.2.0-py3-none-any.whl.metadata (2.0 kB)
Requirement already satisfied: cffi>=1.12 in /srv/conda/envs/notebook/lib/python3.10/site-packages (from cryptography>=2.0->SecretStorage>=3.2->keyring!=23.9.0,>=20.0->datalad->repo2data) (1.17.0)
Requirement already satisfied: pycparser in /srv/conda/envs/notebook/lib/python3.10/site-packages (from cffi>=1.12->cryptography>=2.0->SecretStorage>=3.2->keyring!=23.9.0,>=20.0->datalad->repo2data) (2.22)
Downloading repo2data-2.9.6-py3-none-any.whl (11 kB)
Downloading awscli-1.34.31-py3-none-any.whl (4.5 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.5/4.5 MB 66.9 MB/s eta 0:00:00
Downloading botocore-1.35.31-py3-none-any.whl (12.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.6/12.6 MB 34.4 MB/s eta 0:00:00
Downloading datalad-1.1.3-py3-none-any.whl (1.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 45.2 MB/s eta 0:00:00
Downloading patool-3.0.0-py2.py3-none-any.whl (97 kB)
Downloading gdown-5.2.0-py3-none-any.whl (18 kB)
Downloading osfclient-0.0.5-py2.py3-none-any.whl (39 kB)
Downloading zenodo_get-1.6.1-py2.py3-none-any.whl (17 kB)
Downloading chardet-5.2.0-py3-none-any.whl (199 kB)
Downloading colorama-0.4.6-py2.py3-none-any.whl (25 kB)
Downloading docutils-0.16-py2.py3-none-any.whl (548 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 548.2/548.2 kB 38.9 MB/s eta 0:00:00
Downloading fasteners-0.19-py3-none-any.whl (18 kB)
Downloading keyring-25.4.1-py3-none-any.whl (38 kB)
Downloading rsa-4.7.2-py3-none-any.whl (34 kB)
Downloading s3transfer-0.10.2-py3-none-any.whl (82 kB)
Downloading tqdm-4.66.5-py3-none-any.whl (78 kB)
Downloading annexremote-1.6.6-py3-none-any.whl (25 kB)
Downloading boto3-1.35.31-py3-none-any.whl (139 kB)
Downloading distro-1.9.0-py3-none-any.whl (20 kB)
Downloading filelock-3.16.1-py3-none-any.whl (16 kB)
Downloading humanize-4.10.0-py3-none-any.whl (126 kB)
Downloading iso8601-2.1.0-py3-none-any.whl (7.5 kB)
Downloading keyrings.alt-5.0.2-py3-none-any.whl (17 kB)
Downloading looseversion-1.3.0-py2.py3-none-any.whl (8.2 kB)
Downloading msgpack-1.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (378 kB)
Downloading python_gitlab-4.12.2-py3-none-any.whl (145 kB)
Downloading jeepney-0.8.0-py3-none-any.whl (48 kB)
Downloading jmespath-1.0.1-py3-none-any.whl (20 kB)
Downloading pyasn1-0.6.1-py3-none-any.whl (83 kB)
Downloading requests_toolbelt-1.0.0-py2.py3-none-any.whl (54 kB)
Downloading SecretStorage-3.3.3-py3-none-any.whl (15 kB)
Downloading jaraco.classes-3.4.0-py3-none-any.whl (6.8 kB)
Downloading jaraco.context-6.0.1-py3-none-any.whl (6.8 kB)
Downloading jaraco.functools-4.1.0-py3-none-any.whl (10 kB)
Downloading backports.tarfile-1.2.0-py3-none-any.whl (30 kB)
Downloading more_itertools-10.5.0-py3-none-any.whl (60 kB)
Building wheels for collected packages: wget
Building wheel for wget (setup.py) ... done
Created wheel for wget: filename=wget-3.2-py3-none-any.whl size=9655 sha256=0d2bb4cc28cdbbce5fae9bd2f4c1f60bcdbaba5ccc9809c79e48f5e46d6f2a81
Stored in directory: /home/jovyan/.cache/pip/wheels/8b/f1/7f/5c94f0a7a505ca1c81cd1d9208ae2064675d97582078e6c769
Successfully built wget
Installing collected packages: wget, looseversion, annexremote, tqdm, pyasn1, patool, msgpack, more-itertools, jmespath, jeepney, iso8601, humanize, filelock, fasteners, docutils, distro, colorama, chardet, backports.tarfile, zenodo-get, rsa, requests-toolbelt, osfclient, jaraco.functools, jaraco.context, jaraco.classes, botocore, SecretStorage, s3transfer, python-gitlab, keyrings.alt, gdown, keyring, boto3, awscli, datalad, repo2data
Successfully installed SecretStorage-3.3.3 annexremote-1.6.6 awscli-1.34.31 backports.tarfile-1.2.0 boto3-1.35.31 botocore-1.35.31 chardet-5.2.0 colorama-0.4.6 datalad-1.1.3 distro-1.9.0 docutils-0.16 fasteners-0.19 filelock-3.16.1 gdown-5.2.0 humanize-4.10.0 iso8601-2.1.0 jaraco.classes-3.4.0 jaraco.context-6.0.1 jaraco.functools-4.1.0 jeepney-0.8.0 jmespath-1.0.1 keyring-25.4.1 keyrings.alt-5.0.2 looseversion-1.3.0 more-itertools-10.5.0 msgpack-1.1.0 osfclient-0.0.5 patool-3.0.0 pyasn1-0.6.1 python-gitlab-4.12.2 repo2data-2.9.6 requests-toolbelt-1.0.0 rsa-4.7.2 s3transfer-0.10.2 tqdm-4.66.5 wget-3.2 zenodo-get-1.6.1
jovyan@jupyter-qmrlab-2dmooc-2ddata-2dnczdjebs:~$ repo2data -r data_requirement.json
Traceback (most recent call last):
File "/srv/conda/envs/notebook/bin/repo2data", line 5, in <module>
from repo2data.bin.repo2data import main
ModuleNotFoundError: No module named 'repo2data.bin'
jovyan@jupyter-qmrlab-2dmooc-2ddata-2dnczdjebs:~$
2.9.1 works further on MyBinder, but it's not clear to me if the other error I get is just because of the broken OSF feature for repo2data (I'll just switch to github release files, I know those have worked better before). So this PR really shouldn't be personally a rush for me, but more of a heads up for you @agahkarakuzu if you needed to use the new features you pushed in the latest release(s)
Thanks Mathieu. Recently, I've been using repo2data through python more, as I deprecated its use through terminal within the neurolibre workflow. Do you run into the same issue when you access through python?
There are some redundancies in the implementation logic (no need for external libraries when wget can simply download) and design (e.g., Repo2DataChildren
class that is not actually a subclass of Repo2Data etc.). I'll get to them at some point, which should deal with some of these problems.
Thanks Mathieu. Recently, I've been using repo2data through python more, as I deprecated its use through terminal within the neurolibre workflow. Do you run into the same issue when you access through python?
I've not tried it - that's a great suggestion! If I encounter anything prohibiting me further I'll try that, thanks!
'll get to them at some point, which should deal with some of these problems.
No worries, I think this is pretty low priority considering everything.
Project fresh installed via pip
pip freeze:
repo2data==2.9.6