errors from `datalad install`

[yibeichan]

Hello hello,

I used datalad install -r https://github.com/courtois-neuromod/friends.git to download the friends dataset (I used the HTTP instead of SSH because of some other errors) Then I got the following errors, full errors see here—friends_debug_info.txt

get(error): sub-06/ses-054/anat/sub-06_ses-054_localizer__ch-12_dup-01.nii.gz (file) [not available; (Note that these git remotes have annex-ignore set: origin)]
get(error): sub-06/ses-054/anat/sub-06_ses-054_localizer__ch-23_dup-01.nii.gz (file) [not available; (Note that these git remotes have annex-ignore set: origin)]
get(error): sub-06/ses-054/anat/sub-06_ses-054_localizer__ch-25_dup-01.nii.gz (file) [not available; (Note that these git remotes have annex-ignore set: origin)]
get(error): sub-06/ses-054/anat/sub-06_ses-054_scout.nii.gz (file) [not available; (Note that these git remotes have annex-ignore set: origin)]
get(ok): sub-06/ses-054/fmap/sub-06_ses-054_acq-bold_dir-AP_run-01_epi.nii.gz (file) [from s3unf.cneuromod.friends.mri...]
get(ok): sub-06/ses-054/fmap/sub-06_ses-054_acq-sbref_dir-AP_run-01_epi.nii.gz (file) [from s3unf.cneuromod.friends.mri...]
get(ok): sub-06/ses-054/fmap/sub-06_ses-054_acq-sbref_dir-PA_run-01_epi.nii.gz (file) [from s3unf.cneuromod.friends.mri...]
get(ok): sub-06/ses-054/fmap/sub-06_ses-054_acq-sbref_dir-PA_run-02_epi.nii.gz (file) [from s3unf.cneuromod.friends.mri...]
get(ok): sub-06/ses-054/fmap/sub-06_ses-054_acq-sbref_dir-PA_run-03_epi.nii.gz (file) [from s3unf.cneuromod.friends.mri...]
get(ok): sub-06/ses-054/func/sub-06_ses-054_task-s06e24a_bold.nii.gz (file) [from s3unf.cneuromod.friends.mri...]
get(ok): sub-06/ses-054/func/sub-06_ses-054_task-s06e24a_physio.tsv.gz (file) [from s3unf.cneuromod.friends.mri...]
get(ok): sub-06/ses-054/func/sub-06_ses-054_task-s06e24a_sbref.nii.gz (file) [from s3unf.cneuromod.friends.mri...]
get(ok): sub-06/ses-054/func/sub-06_ses-054_task-s06e24b_bold.nii.gz (file) [from s3unf.cneuromod.friends.mri...]
get(ok): sub-06/ses-054/func/sub-06_ses-054_task-s06e24b_physio.tsv.gz (file) [from s3unf.cneuromod.friends.mri...]
  [4 similar messages have been suppressed; disable with datalad.ui.suppress-similar-results=off]
get(error): sub-06/ses-054/sub-06_ses-054_scans.tsv (file) [not available; (Note that these git remotes have annex-ignore set: origin)]
get(error): sub-06/ses-055/anat/sub-06_ses-055_localizer1.nii.gz (file) [not available; (Note that these git remotes have annex-ignore set: origin)]
get(error): sub-06/ses-055/anat/sub-06_ses-055_localizer2.nii.gz (file) [not available; (Note that these git remotes have annex-ignore set: origin)]
get(error): sub-06/ses-055/anat/sub-06_ses-055_localizer3.nii.gz (file) [not available; (Note that these git remotes have annex-ignore set: origin)]
get(error): sub-06/ses-055/anat/sub-06_ses-055_localizer__ch-55_dup-01.nii.gz (file) [not available; (Note that these git remotes have annex-ignore set: origin)]
get(error): sub-06/ses-055/anat/sub-06_ses-055_localizer__ch-65_dup-01.nii.gz (file) [not available; (Note that these git remotes have annex-ignore set: origin)]
get(error): sub-06/ses-055/anat/sub-06_ses-055_localizer__ch-82_dup-01.nii.gz (file) [not available; (Note that these git remotes have annex-ignore set: origin)]
get(error): sub-06/ses-055/anat/sub-06_ses-055_scout.nii.gz (file) [not available; (Note that these git remotes have annex-ignore set: origin)]
get(ok): sub-06/ses-055/fmap/sub-06_ses-055_acq-bold_dir-AP_run-01_epi.nii.gz (file) [from s3unf.cneuromod.friends.mri...]
get(ok): sub-06/ses-055/fmap/sub-06_ses-055_acq-sbref_dir-AP_run-01_epi.nii.gz (file) [from s3unf.cneuromod.friends.mri...]
get(ok): sub-06/ses-055/fmap/sub-06_ses-055_acq-sbref_dir-PA_run-01_epi.nii.gz (file) [from s3unf.cneuromod.friends.mri...]
get(ok): sub-06/ses-055/func/sub-06_ses-055_task-s06e24d_bold.nii.gz (file) [from s3unf.cneuromod.friends.mri...]
get(ok): sub-06/ses-055/func/sub-06_ses-055_task-s06e24d_physio.tsv.gz (file) [from s3unf.cneuromod.friends.mri...]
get(ok): sub-06/ses-055/func/sub-06_ses-055_task-s06e24d_sbref.nii.gz (file) [from s3unf.cneuromod.friends.mri...]
get(error): sub-06/ses-055/sub-06_ses-055_scans.tsv (file) [not available; (Note that these git remotes have annex-ignore set: origin)]
action summary:
  get (error: 2607, ok: 7573)

I'm afraid those errors will fail fmriprep we plan to run. I noticed there are scan issues on this repository. Are those scan issues related to the above error messages we got?

also, I remember seeing errors from datalad installing friends-stimuli subdataset.

thank you for your help!

[pbellec]

Thanks for reporting.

As indicated in the documentation it is recommended to use ssh. Please share your ssh error logs.

Also note that you need a key to access some of the data, while some other are fully open. So unless you have installed the S3 access key you will get errors but the open part of the dataset will still get installed.

Basile our data manager is off today but we'll look into your issues if any remain tomorrow.

[pbellec]

Another point: running friends through fmriprep is not a standard process. friends do not contain T1w maps. Those are shared in the anat datasets and include about 15 sessions that have to be combined to produce high quality T1w individual maps and registration. The Cneuromod team shares all derivatives already.

[yibeichan]

Hi thank you! I did use S3 access key So I tried the ssh one again; the error is related to missing ssh key linking the server with my github account; I fixed it and test datalad install -r git@github.com:courtois-neuromod/hcptrt.git. It works well. I guess we can close this issue.

so, for friends, should we use https://github.com/courtois-neuromod/friends.fmriprep directly?

[pbellec]

Great to hear it worked! Yes I would advise to use the fmriprep derivatives provided by our team.

If for some reason you are interested in running fMRIprep yourself (e.g. to use a different version), I suggest we get in touch and discuss necessary adjustments to a standard workflow.