Test dowload of subject in ConnectomeDB

[oportoles]

I went through the project is the connetomeDB and try to obtain the data of a subject

The process was datalad create abcd cd abcd datalad xnat-init
datalad xnat-init https://db.humanconnectome.org --project HCP_Retest -f datalad xnat-update -s ConnectomeDB_S02330 datalad xnat-init https://db.humanconnectome.org --project HCP_1200

if has subjects datalad xnat-update -s XXX

MGH_DIFF project works

project HCP_1200 $ datalad xnat-update -s ConnectomeDB_S00575 [INFO ] Querying info for subject ConnectomeDB_S00575 [ERROR ] JSONDecodeError(Expecting value: line 1 column 1 (char 0)) (JSONDecodeError)

$ datalad xnat-update -s ConnectomeDB_S02347 [INFO ] Querying info for subject ConnectomeDB_S02347 [ERROR ] JSONDecodeError(Expecting value: line 1 column 1 (char 0)) (JSONDecodeError)

Project HCP_500 has no subject Project HCP_500_RST has no subject Project CCF_DMCC_STG has no subjects Project HCP_900 has no subject Project HCP_Coded has no subject Project HCP_Q1 has no subject Project HCP_Q2 has no subject Project HCP_Q2 has no subject Project HCP_Q3 has no subject Project HCP_Q1 has no subject Project HCP_Q3_RST has no subject Project HCP_Resources has no subject Project HCP_Restricted has no subject Project HCP_Staging_RT has no subject

Project HCP_Subjects datalad xnat-update -s ConnectomeDB_S02346 [ERROR ] JSONDecodeError(Expecting value: line 1 column 1 (char 0)) (JSONDecodeError)

Project HCP_Subjects_R datalad xnat-update -s ConnectomeDB_S02330
[ERROR ] JSONDecodeError(Expecting value: line 1 column 1 (char 0)) (JSONDecodeError)

Project HCP_TFMRI has no subject Project MGH_DIFF has no subject Project PublicProject has no subject Project RESTRICT_TIER1 has no subject Project WU_L1A_Cur works ConnectomeDB_S01439 $ datalad xnat-update ConnectomeDB_S01439

Project WU_L1A_Cur_RST has no subjects Project WU_L1A_Restr has no subjects Project WU_L1A_Staging has no subjects

Project WU_L1A_Subj datalad xnat-update -s ConnectomeDB_S01439 [ERROR ] JSONDecodeError(Expecting value: line 1 column 1 (char 0)) (JSONDecodeError)

Project WU_L1A_Unproc has no subjects Project WU_L1B_Staging has no subjects Project WU_L1B_Subj has no subjects

[adswa]

"Getting a project from ConnectomeDB" would be a cool tutorial use case for the docs, I think. I've restructured the documentation in #89 so that it is more user-oriented, with a -- albeit currently yet empty -- tutorial section. A walkthrough on getting any random connectome DB dataset (once functional) would be cool because anyone can get credentials for this to try it out themselves.

[adswa]

hey @all-contributors please add @oportoles for userTesting

[allcontributors[bot]]

@adswa

I've put up a pull request to add @oportoles! :tada:

[bpoldrack]

FTR: With PR #81 merged, the JSONDecodeErrors you have been seeing, @oportoles, should now be more informative and point to how the requests failed. Would be nice to see that confirmed.

[oportoles]

FTR: With PR #81 merged, the JSONDecodeErrors you have been seeing, @oportoles, should now be more informative and point to how the requests failed. Would be nice to see that confirmed.

I have run the same test with the new version on the master branch that includes the PR #81 @bpoldrack

Now, the previously working projects are failing to download a subject. The previously failing projects are still failing. The error messages has become more informative in some cases. First error was that ipython ws missing. After installing ipython the error were pointed at KeyError: 'digest' sometimes. Below are the steps that I followed after I finished the previous test

$ git pull $ datalad xnat-update --version datalad_xnat 0+untagged.257.g6fd301f

datalad xnat-init https://db.humanconnectome.org

Previously faling project

datalad xnat-init https://db.humanconnectome.org -p WU_L1A_Subj -f datalad xnat-update -s ConnectomeDB_S01439 Failed to obtain information on subject ConnectomeDB_S01439 from XNAT project WU_L1A_Subj:

datalad --dbg xnat-update -s ConnectomeDB_S01439 Failed to obtain information on subject ConnectomeDB_S01439 from XNAT project WU_L1A_Subj:

datalad --idbg xnat-update -s ConnectomeDB_S01439 Traceback (most recent call last): File "/home/oportoles/miniconda3/envs/hacka-dl-xnat/bin/datalad", line 33, in sys.exit(load_entry_point('datalad', 'console_scripts', 'datalad')()) File "/home/oportoles/Documents/MyCode/dl_hackathon/datalad/datalad/cmdline/main.py", line 202, in main setup_exceptionhook(ipython=cmdlineargs.common_idebug) File "/home/oportoles/Documents/MyCode/dl_hackathon/datalad/datalad/utils.py", line 1498, in setup_exceptionhook from IPython.core import ultratb ModuleNotFoundError: No module named 'IPython'

conda install -c conda-forge ipython

New terminal

Try a previously working project

datalad xnat-init https://db.humanconnectome.org --project MGH_DIFF -f datalad xnat-update datalad xnat-update -s ConnectomeDB_S01356 [INFO ] Querying info for subject ConnectomeDB_S01356 [ERROR ] KeyError('digest') (KeyError)

datalad --dbg xnat-update -s ConnectomeDB_S01356
[INFO ] Querying info for subject ConnectomeDB_S01356 Traceback (most recent call last): File "/home/oportoles/miniconda3/envs/hacka-dl-xnat/bin/datalad", line 33, in sys.exit(load_entry_point('datalad', 'console_scripts', 'datalad')()) File "/home/oportoles/Documents/MyCode/dl_hackathon/datalad/datalad/cmdline/main.py", line 205, in main ret = cmdlineargs.func(cmdlineargs) File "/home/oportoles/Documents/MyCode/dl_hackathon/datalad/datalad/interface/base.py", line 785, in call_from_parser ret = list(ret) File "/home/oportoles/Documents/MyCode/dl_hackathon/datalad/datalad/interface/utils.py", line 396, in generator_func for r in _process_results( File "/home/oportoles/Documents/MyCode/dl_hackathon/datalad/datalad/interface/utils.py", line 574, in _process_results for res in results: File "/home/oportoles/Documents/MyCode/hacka_datalad-xnat/datalad-xnat/datalad_xnat/update.py", line 189, in call yield from parse_xnat( File "/home/oportoles/Documents/MyCode/hacka_datalad-xnat/datalad-xnat/datalad_xnat/parser.py", line 42, in parse_xnat for fr in query_files(platform, project=xnat_project, subject=sub): File "/home/oportoles/Documents/MyCode/hacka_datalad-xnat/datalad-xnat/datalad_xnat/query_files.py", line 158, in query_files digest = fr.pop('digest') KeyError: 'digest'

/home/oportoles/Documents/MyCode/hacka_datalad-xnat/datalad-xnat/datalad_xnat/query_files.py(158)query_files() -> digest = fr.pop('digest') (Pdb) q

Try another project that reviusly worked: WU_L1A_Cur

datalad xnat-update -s ConnectomeDB_S01356 datalad xnat-update datalad xnat-update -s ConnectomeDB_S01439
Failed to obtain information on subject ConnectomeDB_S01439 from XNAT project WU_L1A_Cur:

datalad --dbg xnat-update -s ConnectomeDB_S01439 Failed to obtain information on subject ConnectomeDB_S01439 from XNAT project WU_L1A_Cur:

oportoles@bnbnb61 in ~/Documents/MyCode/dl_hackathon/abcd on git:master datalad --idbg xnat-update -s ConnectomeDB_S01439 Failed to obtain information on subject ConnectomeDB_S01439 from XNAT project WU_L1A_Cur:

Try project that gave error before: HCP_Subjects_R datalad xnat-init https://db.humanconnectome.org --project HCP_Subjects_R -f datalad xnat-update
datalad xnat-update -s ConnectomeDB_S02330 Failed to obtain information on subject ConnectomeDB_S02330 from XNAT project HCP_Subjects_R:

datalad --dbg xnat-update -s ConnectomeDB_S02330 Failed to obtain information on subject ConnectomeDB_S02330 from XNAT project HCP_Subjects_R: datalad --idbg xnat-update -s ConnectomeDB_S02330 Failed to obtain information on subject ConnectomeDB_S02330 from XNAT project HCP_Subjects_R:

[oportoles]

Project MGH_DIFF gave the error KeyError: 'digest' and after #124 can download a subject correctly. Other projects that worked on first place are still giving an error

git pull datalad xnat-update --version datalad_xnat 0+untagged.268.g712b3c7

❱ datalad xnat-init https://db.humanconnectome.org ❱ datalad xnat-init https://db.humanconnectome.org --project MGH_DIFF -f ❱ datalad xnat-update ❱ datalad xnat-update -s ConnectomeDB_S01356 [INFO ] Querying info for subject ConnectomeDB_S01356 xnat_query(ok): ConnectomeDB_E05060/10/T1.nii.gz (file) [NIFTI] ...

Project download that gave error after #81, but worked initially, still gives error

❱ datalad xnat-init https://db.humanconnectome.org --project WU_L1A_Cur -f ❱ datalad xnat-update ❱ datalad xnat-update -s ConnectomeDB_S01439
Failed to obtain information on subject ConnectomeDB_S01439 from XNAT project WU_L1A_Cur: oportoles@bnbnb61 in ~/Documents/MyCode/dl_hackathon/abcd on git:master ❱ datalad --dbg xnat-update -s ConnectomeDB_S01439 Failed to obtain information on subject ConnectomeDB_S01439 from XNAT project WU_L1A_Cur:

❱ datalad --idbg xnat-update -s ConnectomeDB_S01439 Failed to obtain information on subject ConnectomeDB_S01439 from XNAT project WU_L1A_Cur:

[mih]

I think most errors here are fixed, or are related to insufficient permissions to access particular files.