Closed jkkenley closed 1 year ago
Hi Jeanette! Thanks for submitting your error to us. Someone from our lab will be in contact with you as soon as we are able.
Hi Jeanette,
My best guess is the error is unrelated to dataset_description.json and the actual problem is the "EffectiveEchoSpacing" field is missing from the fmap json files. (I believe older versions of Dcm2bids do not output EffectiveEchoSpacing, so that may be the cause.) Can you check:
1) If "EffectiveEchoSpacing" is present in the fmap directory json files 2) If the json files have infomation about the Dcm2bids or dcm2niix version used to convert from DICOM (for Dcm2bids the field is titled "Dcm2bidsVersion")
Thanks!
(edit to add: assuming EffectiveEchoSpacing is the problem, as long as you other datasets used the same fmap sequence it would be fine to copy the EffectiveEchoSpacing info from those to the problem subject, then try rerunning it.)
1.) "BandwidthPerPixelPhaseEncode": 21.786, "EffectiveEchoSpacing": 0.000510012, "DerivedVendorReportedEchoSpacing": 0.000510012, "TotalReadoutTime": 0.045391,
2.) "InPlanePhaseEncodingDirectionDICOM": "COL", "ConversionSoftware": "dcm2niix", "ConversionSoftwareVersion": "v1.0.20210317", "Dcm2bidsVersion": "2.1.6"
These parameters are consistet with a subject who did run.
Thanks, Jeanette
The materials in this email are private and may contain Protected Health Information. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying, distribution or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return email.
From: Thomas Madison @.> Sent: Thursday, May 12, 2022 6:02 PM To: DCAN-Labs/abcd-hcp-pipeline @.> Cc: Kenley, Jeanette @.>; Author @.> Subject: Re: [DCAN-Labs/abcd-hcp-pipeline] dataset_description.json (Issue #79)
Hi Jeanette,
My best guess is the error is unrelated to dataset_description.json and the actual problem is the "EffectiveEchoSpacing" field is missing from the fmap json files. (I believe older versions of Dcm2bids do not output EffectiveEchoSpacing, so that may be the cause.) Can you check:
Thanks!
— Reply to this email directly, view it on GitHubhttps://github.com/DCAN-Labs/abcd-hcp-pipeline/issues/79#issuecomment-1125493627, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AUYL7WTMY6AHBWT4YRT3M7TVJWE2HANCNFSM5VVKRNEA. You are receiving this because you authored the thread.Message ID: @.***>
The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.
Hi Jeanette,
Apologies for the slow response. Thank you for posting that info, that rules out the EffectiveEchoSpacing issue which was my first guess.
Could you share:
1) the BIDS directory tree for the subject (removing subject and session IDs is OK) 2) the command or script used to run the pipeline
Thank you!
I have to send it in parts because of the cleaning file.
Thank you for all your help! Jeanette
The materials in this email are private and may contain Protected Health Information. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying, distribution or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return email.
From: Thomas Madison @.> Sent: Monday, May 23, 2022 1:26 PM To: DCAN-Labs/abcd-hcp-pipeline @.> Cc: Kenley, Jeanette @.>; Author @.> Subject: Re: [DCAN-Labs/abcd-hcp-pipeline] dataset_description.json (Issue #79)
Hi Jeanette,
Apologies for the slow response. Thank you for posting that info, that rules out the EffectiveEchoSpacing issue which was my first guess.
Could you share:
Thank you!
— Reply to this email directly, view it on GitHubhttps://github.com/DCAN-Labs/abcd-hcp-pipeline/issues/79#issuecomment-1135001127, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AUYL7WXDWTMO4LW7F2AYDFDVLPEWHANCNFSM5VVKRNEA. You are receiving this because you authored the thread.Message ID: @.***>
The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.
patid/sub-patid/func: total 767990 -rw-r--r--+ 1 kenleyj uucp 196236721 May 11 08:39 sub-patid_task-rest_run-04_bold.nii.gz -rw-r--r--+ 1 kenleyj uucp 195881661 May 11 08:40 sub-patid_task-rest_run-03_bold.nii.gz -rw-r--r--+ 1 kenleyj uucp 200983437 May 11 08:40 sub-patid_task-rest_run-01_bold.nii.gz -rw-r--r--+ 1 kenleyj uucp 192768334 May 11 08:41 sub-patid_task-rest_run-02_bold.nii.gz -rw-r--r--+ 1 kenleyj uucp 3124 May 11 08:41 sub-patid_task-rest_run-04_bold.json -rw-r--r--+ 1 kenleyj uucp 3094 May 11 08:41 sub-patid_task-rest_run-03_bold.json -rw-r--r--+ 1 kenleyj uucp 3093 May 11 08:41 sub-patid_task-rest_run-02_bold.json -rw-r--r--+ 1 kenleyj uucp 3112 May 11 08:41 sub-patid_task-rest_run-01_bold.json
patid/sub-patid/fmap: total 2636 -rw-r--r--+ 1 kenleyj uucp 647673 May 11 08:39 sub-patid_dir-PA_run-01_epi.nii.gz -rw-r--r--+ 1 kenleyj uucp 643439 May 11 08:39 sub-patid_dir-AP_run-01_epi.nii.gz -rw-r--r--+ 1 kenleyj uucp 628766 May 11 08:41 sub-patid_dir-PA_run-02_epi.nii.gz -rw-r--r--+ 1 kenleyj uucp 628950 May 11 08:41 sub-patid_dir-AP_run-02_epi.nii.gz -rw-r--r--+ 1 kenleyj uucp 3178 May 11 08:41 sub-patid_dir-PA_run-01_epi.json -rw-r--r--+ 1 kenleyj uucp 3172 May 11 08:41 sub-patid_dir-AP_run-01_epi.json -rw-r--r--+ 1 kenleyj uucp 3173 May 11 08:41 sub-patid_dir-PA_run-02_epi.json -rw-r--r--+ 1 kenleyj uucp 3178 May 11 08:41 sub-patid_dir-AP_run-02_epi.json
patid/sub-patid/anat: total 24718 -rw-r--r--+ 1 kenleyj uucp 14178622 May 11 08:39 sub-patid_T1w.nii.gz -rw-r--r--+ 1 kenleyj uucp 11038105 May 11 08:41 sub-patid_T2w.nii.gz -rw-r--r--+ 1 kenleyj uucp 1969 May 11 08:41 sub-patid_T1w.json -rw-r--r--+ 1 kenleyj uucp 2000 May 11 08:41 sub-patid_T2w.json
@jkkenley - Were you able to send this over to Thomas? If you no longer need assistance, feel free to close this issue. :)
@Amanda @.***>/Thomas
I tried sending the cleaning file several times, but it kept getting kicked back. Trying a different approach now. Thomas, thank you for your help.
Jeanette
The materials in this email are private and may contain Protected Health Information. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying, distribution or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return email.
From: Amanda Rueter @.> Sent: Wednesday, June 22, 2022 3:25 PM To: DCAN-Labs/abcd-hcp-pipeline @.> Cc: Kenley, Jeanette @.>; Mention @.> Subject: Re: [DCAN-Labs/abcd-hcp-pipeline] dataset_description.json (Issue #79)
@jkkenleyhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fjkkenley&data=05%7C01%7Cjkkenley%40wustl.edu%7C24ddb15c26564c8b9c8f08da548d66df%7C4ccca3b571cd4e6d974b4d9beb96c6d6%7C0%7C0%7C637915263540539252%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=vPPb8BM0%2FwgVuxo7B1UQLf7kaA6vZfJv7eWWrKryohI%3D&reserved=0 - Were you able to send this over to Thomas? If you no longer need assistance, feel free to close this issue. :)
— Reply to this email directly, view it on GitHubhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FDCAN-Labs%2Fabcd-hcp-pipeline%2Fissues%2F79%23issuecomment-1163566619&data=05%7C01%7Cjkkenley%40wustl.edu%7C24ddb15c26564c8b9c8f08da548d66df%7C4ccca3b571cd4e6d974b4d9beb96c6d6%7C0%7C0%7C637915263540539252%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=vk7I4Vx4dTJMCfXfBrUGcyHR%2FyeG9Dx4GeQ4mk9kukQ%3D&reserved=0, or unsubscribehttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAUYL7WSW5SEWH6KYYMVQ6CDVQNZE3ANCNFSM5VVKRNEA&data=05%7C01%7Cjkkenley%40wustl.edu%7C24ddb15c26564c8b9c8f08da548d66df%7C4ccca3b571cd4e6d974b4d9beb96c6d6%7C0%7C0%7C637915263540539252%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Y6ft13PzXfk2TPW2qG1%2FuSMkZKxEAnNXYb4EkULFEzM%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.***>
The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.
I came across this thread while searching a similar error message. For my data, it turned out that one of my field map json files was missing the IntendedFor entry. I was confused because when I compared my AP fmap json to a known good subject everything looked fine (similar to the above user), but my PA fmap was missing the IntendedFor entry. Once I corrected that it (so both the AP and PA fmap json had the correct IntendedFor directive) then it resolved the error. Hope this is helpful.
Thanks so much! I'll give this a try!
The materials in this email are private and may contain Protected Health Information. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying, distribution or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return email.
From: Jarod @.> Sent: Wednesday, September 14, 2022 5:57 PM To: DCAN-Labs/abcd-hcp-pipeline @.> Cc: Kenley, Jeanette @.>; Mention @.> Subject: Re: [DCAN-Labs/abcd-hcp-pipeline] dataset_description.json (Issue #79)
I came across this thread while searching a similar error message. For my data, it turned out that one of my field map json files was missing the IntendedFor entry. I was confused because when I compared my AP fmap json to a known good subject everything looked fine (similar to the above user), but my PA fmap was missing the IntendedFor entry. Once I corrected that it (so both the AP and PA fmap json had the correct IntendedFor directive) then it resolved the error. Hope this is helpful.
— Reply to this email directly, view it on GitHubhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FDCAN-Labs%2Fabcd-hcp-pipeline%2Fissues%2F79%23issuecomment-1247382706&data=05%7C01%7Cjkkenley%40wustl.edu%7Cce35f62c582441fd21df08da96a48356%7C4ccca3b571cd4e6d974b4d9beb96c6d6%7C0%7C0%7C637987930570302957%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=zt86symem2LGPMEwPwNN9BF0zSzihFqZk4djeX7ECuI%3D&reserved=0, or unsubscribehttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAUYL7WVJ62RJTHE555QEU6DV6JJ5ZANCNFSM5VVKRNEA&data=05%7C01%7Cjkkenley%40wustl.edu%7Cce35f62c582441fd21df08da96a48356%7C4ccca3b571cd4e6d974b4d9beb96c6d6%7C0%7C0%7C637987930570302957%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=CLOQkYYlrKhJftQb3lF216yybLnW2QNtb3S59aWWmZc%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.***>
The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.
Any further updates on this by chance?
The materials in this email are private and may contain Protected Health Information. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying, distribution or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return email.
From: Kenley, Jeanette @.> Sent: Friday, May 13, 2022 8:28 AM To: DCAN-Labs/abcd-hcp-pipeline @.>; DCAN-Labs/abcd-hcp-pipeline @.> Cc: Author @.> Subject: Re: [DCAN-Labs/abcd-hcp-pipeline] dataset_description.json (Issue #79)
1.) "BandwidthPerPixelPhaseEncode": 21.786, "EffectiveEchoSpacing": 0.000510012, "DerivedVendorReportedEchoSpacing": 0.000510012, "TotalReadoutTime": 0.045391,
2.) "InPlanePhaseEncodingDirectionDICOM": "COL", "ConversionSoftware": "dcm2niix", "ConversionSoftwareVersion": "v1.0.20210317", "Dcm2bidsVersion": "2.1.6"
These parameters are consistet with a subject who did run.
Thanks, Jeanette
The materials in this email are private and may contain Protected Health Information. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying, distribution or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return email.
From: Thomas Madison @.> Sent: Thursday, May 12, 2022 6:02 PM To: DCAN-Labs/abcd-hcp-pipeline @.> Cc: Kenley, Jeanette @.>; Author @.> Subject: Re: [DCAN-Labs/abcd-hcp-pipeline] dataset_description.json (Issue #79)
Hi Jeanette,
My best guess is the error is unrelated to dataset_description.json and the actual problem is the "EffectiveEchoSpacing" field is missing from the fmap json files. (I believe older versions of Dcm2bids do not output EffectiveEchoSpacing, so that may be the cause.) Can you check:
Thanks!
— Reply to this email directly, view it on GitHubhttps://github.com/DCAN-Labs/abcd-hcp-pipeline/issues/79#issuecomment-1125493627, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AUYL7WTMY6AHBWT4YRT3M7TVJWE2HANCNFSM5VVKRNEA. You are receiving this because you authored the thread.Message ID: @.***>
The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.
Hi @jkkenley, were you able to successfully provide @madisoth with what he was looking for? (the BIDS directory tree for the subject, the command or script used to run the pipeline)
I am having the same issue described above except that I'm testing running the pipeline on two participants, and it happens for both. Here's the directory structure: sub-001 sub-001/ses-06 sub-001/ses-06/anat sub-001/ses-06/anat/sub-001_ses-06_T1w.json sub-001/ses-06/anat/sub-001_ses-06_T1w.nii.gz sub-001/ses-06/dwi sub-001/ses-06/dwi/sub-001_ses-06_dir-PA_dwi.bval sub-001/ses-06/dwi/sub-001_ses-06_dir-PA_dwi.json sub-001/ses-06/dwi/sub-001_ses-06_dir-AP_dwi.bval sub-001/ses-06/dwi/sub-001_ses-06_dir-AP_dwi.bvec sub-001/ses-06/dwi/sub-001_ses-06_dir-AP_dwi.json sub-001/ses-06/dwi/sub-001_ses-06_dir-PA_dwi.bvec sub-001/ses-06/dwi/sub-001_ses-06_dir-PA_dwi.nii.gz sub-001/ses-06/dwi/sub-001_ses-06_dir-AP_dwi.nii.gz sub-001/ses-06/fmap sub-001/ses-06/fmap/sub-001_ses-06_acq-rest_dir-AP_epi.json sub-001/ses-06/fmap/sub-001_ses-06_acq-hariri_dir-AP_epi.json sub-001/ses-06/fmap/sub-001_ses-06_acq-hariri_dir-AP_epi.nii.gz sub-001/ses-06/fmap/sub-001_ses-06_acq-rest_dir-AP_epi.nii.gz sub-001/ses-06/func sub-001/ses-06/func/sub-001_ses-06_task-rest_bold.nii.gz sub-001/ses-06/func/sub-001_ses-06_task-rest_bold.json sub-001/ses-06/func/sub-001_ses-06_task-hariri_bold.nii.gz sub-001/ses-06/func/sub-001_ses-06_task-hariri_bold.json sub-001/ses-06/func/sub-001_ses-06_task-hariri_events.tsv sub-001/ses-06/sub-001_ses-06_scans.json sub-001/ses-06/sub-001_ses-06_scans.tsv
EffectiveEchoSpacing is in the fmap jsons, and they all have IntendedFor in the relevant fmap json.
I've run it via as an sbatch script that runs the following command: env -i singularity run \ -B /PATH/TO/DATA:/bids_input \ -B /PATH/TO/OUTPUT:/output \ -B /PATH/TO/license.txt:/opt/freesurfer/license.txt \ ./abcd-hcp-pipeline.sif /bids_input /output --freesurfer-license=/opt/freesurfer/license.txt --participant-label 001 --session-id 06
Using version: abcd-hcp-pipeline 0.1.2
Here are the resources I requested when submitting the job:
I also loaded the following modules: module load fsl module load python3 module load freesurfer module load singularity
Appreciate the help!
@fpichardo-umn
The pipeline will assume that since your fmap filenames end in "_epi.nii.gz", you have PEPolar fmaps (e.g. both "dir-AP" and "dir-PA") for topup correction. With only "dir-AP" in the fmap dir an error would be expected.
The fieldmap schemes the pipelines can handle are cases 2 and 4 in the BIDS spec here: https://bids-specification.readthedocs.io/en/stable/04-modality-specific-files/01-magnetic-resonance-imaging-data.html#types-of-fieldmaps
Another potential issue is that the BIDS "acq-" entity in filenames may go unrecognized (it's a known issue with the pipeline), so you may need to rename files from this:
sub-001/ses-06/fmap/sub-001_ses-06_acq-hariri_dir-AP_epi.nii.gz
sub-001/ses-06/fmap/sub-001_ses-06_acq-rest_dir-AP_epi.nii.gz
to this:
sub-001/ses-06/fmap/sub-001_ses-06_dir-AP_run-01_epi.nii.gz
sub-001/ses-06/fmap/sub-001_ses-06_dir-AP_run-02_epi.nii.gz
Also, if running on the MSI HPC system via Singularity container then Singularity should be the only module you need to load, all other dependencies (other than the FreeSurfer license) are built into the container.
This fixed the issue (specifically checking on the fieldmaps and making sure there were two there). Thanks for the help!
I am using the following docker call to run our data through:
docker run --rm --user 1191:10 \ -v /docker_input_2020_09_15:/bids_input:ro \ -v /docker_output_2020_09_15:/output \ -v /usr/local/pkg/freesurfer6.0/license.txt:/license \ -v/subject_specific_code/subID_wcII_ABCD_BIDS_cleaning.json:/cleaning.json \ dcanlabs/abcd-hcp-pipeline:v0.0.3 /bids_input /output --freesurfer-license=/license \ --participant-label subID --ncpus 10 --bandstop 18.582 25.7263 --custom-clean /cleaning.json
I receive the following error when I run this subject but not 3 others: /usr/local/lib/python3.6/dist-packages/bids/layout/bids_layout.py:116: UserWarning: 'dataset_description.json' file is missing from project root. You may want to set the root path to a valid BIDS project. warnings.warn("'dataset_description.json' file is missing from " Traceback (most recent call last): File "/app/run.py", line 364, in
_cli()
File "/app/run.py", line 67, in _cli
return interface(**kwargs)
File "/app/run.py", line 263, in interface
session_spec = ParameterSettings(session, out_dir)
File "/app/pipelines.py", line 126, in init
self.echospacing = self.bids_data['fmap_metadata']['positive'][0][
IndexError: list index out of range
I have compared this data set to another that ran through and cannot find any obvious failure reasons.
I had gotten this error in the past and was pointed to this site: https://bids-specification.readthedocs.io/en/stable/03-modality-agnostic-files.html#dataset_descriptionjson
But I'm not sure what to do with the information since I had 3 other data sets that ran through.
Thank you for the help. Jeanette