search

DCAN-Labs / abcd-hcp-pipeline

bids application for processing functional MRI data, robust to scanner, acquisition and age variability.
https://hub.docker.com/r/dcanumn/abcd-hcp-pipeline
BSD 3-Clause "New" or "Revised" License
48 stars 19 forks source link

dataset_description.json #79

Closed jkkenley closed 1 year ago

jkkenley commented 2 years ago

I am using the following docker call to run our data through:

docker run --rm --user 1191:10 \ -v /docker_input_2020_09_15:/bids_input:ro \ -v /docker_output_2020_09_15:/output \ -v /usr/local/pkg/freesurfer6.0/license.txt:/license \ -v/subject_specific_code/subID_wcII_ABCD_BIDS_cleaning.json:/cleaning.json \ dcanlabs/abcd-hcp-pipeline:v0.0.3 /bids_input /output --freesurfer-license=/license \ --participant-label subID --ncpus 10 --bandstop 18.582 25.7263 --custom-clean /cleaning.json

I receive the following error when I run this subject but not 3 others: /usr/local/lib/python3.6/dist-packages/bids/layout/bids_layout.py:116: UserWarning: 'dataset_description.json' file is missing from project root. You may want to set the root path to a valid BIDS project. warnings.warn("'dataset_description.json' file is missing from " Traceback (most recent call last): File "/app/run.py", line 364, in _cli() File "/app/run.py", line 67, in _cli return interface(**kwargs) File "/app/run.py", line 263, in interface session_spec = ParameterSettings(session, out_dir) File "/app/pipelines.py", line 126, in init self.echospacing = self.bids_data['fmap_metadata']['positive'][0][ IndexError: list index out of range

I have compared this data set to another that ran through and cannot find any obvious failure reasons.
I had gotten this error in the past and was pointed to this site: https://bids-specification.readthedocs.io/en/stable/03-modality-agnostic-files.html#dataset_descriptionjson

But I'm not sure what to do with the information since I had 3 other data sets that ran through.

Thank you for the help. Jeanette

arueter1 commented 2 years ago

Hi Jeanette! Thanks for submitting your error to us. Someone from our lab will be in contact with you as soon as we are able.

madisoth commented 2 years ago

Hi Jeanette,

My best guess is the error is unrelated to dataset_description.json and the actual problem is the "EffectiveEchoSpacing" field is missing from the fmap json files. (I believe older versions of Dcm2bids do not output EffectiveEchoSpacing, so that may be the cause.) Can you check:

1) If "EffectiveEchoSpacing" is present in the fmap directory json files 2) If the json files have infomation about the Dcm2bids or dcm2niix version used to convert from DICOM (for Dcm2bids the field is titled "Dcm2bidsVersion")

Thanks!

(edit to add: assuming EffectiveEchoSpacing is the problem, as long as you other datasets used the same fmap sequence it would be fine to copy the EffectiveEchoSpacing info from those to the problem subject, then try rerunning it.)

jkkenley commented 2 years ago

1.) "BandwidthPerPixelPhaseEncode": 21.786, "EffectiveEchoSpacing": 0.000510012, "DerivedVendorReportedEchoSpacing": 0.000510012, "TotalReadoutTime": 0.045391,

2.) "InPlanePhaseEncodingDirectionDICOM": "COL", "ConversionSoftware": "dcm2niix", "ConversionSoftwareVersion": "v1.0.20210317", "Dcm2bidsVersion": "2.1.6"

These parameters are consistet with a subject who did run.

Thanks, Jeanette

The materials in this email are private and may contain Protected Health Information. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying, distribution or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return email.

From: Thomas Madison @.> Sent: Thursday, May 12, 2022 6:02 PM To: DCAN-Labs/abcd-hcp-pipeline @.> Cc: Kenley, Jeanette @.>; Author @.> Subject: Re: [DCAN-Labs/abcd-hcp-pipeline] dataset_description.json (Issue #79)

Hi Jeanette,

My best guess is the error is unrelated to dataset_description.json and the actual problem is the "EffectiveEchoSpacing" field is missing from the fmap json files. (I believe older versions of Dcm2bids do not output EffectiveEchoSpacing, so that may be the cause.) Can you check:

  1. If "EffectiveEchoSpacing" is present in the fmap directory json files
  2. If the json files have infomation about the Dcm2bids or dcm2niix version used to convert from DICOM (for Dcm2bids the field is titled "Dcm2bidsVersion")

Thanks!

— Reply to this email directly, view it on GitHubhttps://github.com/DCAN-Labs/abcd-hcp-pipeline/issues/79#issuecomment-1125493627, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AUYL7WTMY6AHBWT4YRT3M7TVJWE2HANCNFSM5VVKRNEA. You are receiving this because you authored the thread.Message ID: @.***>

The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

madisoth commented 2 years ago

Hi Jeanette,

Apologies for the slow response. Thank you for posting that info, that rules out the EffectiveEchoSpacing issue which was my first guess.

Could you share:

1) the BIDS directory tree for the subject (removing subject and session IDs is OK) 2) the command or script used to run the pipeline

Thank you!

jkkenley commented 2 years ago

I have to send it in parts because of the cleaning file.

Thank you for all your help! Jeanette

The materials in this email are private and may contain Protected Health Information. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying, distribution or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return email.

From: Thomas Madison @.> Sent: Monday, May 23, 2022 1:26 PM To: DCAN-Labs/abcd-hcp-pipeline @.> Cc: Kenley, Jeanette @.>; Author @.> Subject: Re: [DCAN-Labs/abcd-hcp-pipeline] dataset_description.json (Issue #79)

Hi Jeanette,

Apologies for the slow response. Thank you for posting that info, that rules out the EffectiveEchoSpacing issue which was my first guess.

Could you share:

  1. the BIDS directory tree for the subject (removing subject and session IDs is OK)
  2. the command or script used to run the pipeline

Thank you!

— Reply to this email directly, view it on GitHubhttps://github.com/DCAN-Labs/abcd-hcp-pipeline/issues/79#issuecomment-1135001127, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AUYL7WXDWTMO4LW7F2AYDFDVLPEWHANCNFSM5VVKRNEA. You are receiving this because you authored the thread.Message ID: @.***>

The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

patid/sub-patid/func: total 767990 -rw-r--r--+ 1 kenleyj uucp 196236721 May 11 08:39 sub-patid_task-rest_run-04_bold.nii.gz -rw-r--r--+ 1 kenleyj uucp 195881661 May 11 08:40 sub-patid_task-rest_run-03_bold.nii.gz -rw-r--r--+ 1 kenleyj uucp 200983437 May 11 08:40 sub-patid_task-rest_run-01_bold.nii.gz -rw-r--r--+ 1 kenleyj uucp 192768334 May 11 08:41 sub-patid_task-rest_run-02_bold.nii.gz -rw-r--r--+ 1 kenleyj uucp 3124 May 11 08:41 sub-patid_task-rest_run-04_bold.json -rw-r--r--+ 1 kenleyj uucp 3094 May 11 08:41 sub-patid_task-rest_run-03_bold.json -rw-r--r--+ 1 kenleyj uucp 3093 May 11 08:41 sub-patid_task-rest_run-02_bold.json -rw-r--r--+ 1 kenleyj uucp 3112 May 11 08:41 sub-patid_task-rest_run-01_bold.json

patid/sub-patid/fmap: total 2636 -rw-r--r--+ 1 kenleyj uucp 647673 May 11 08:39 sub-patid_dir-PA_run-01_epi.nii.gz -rw-r--r--+ 1 kenleyj uucp 643439 May 11 08:39 sub-patid_dir-AP_run-01_epi.nii.gz -rw-r--r--+ 1 kenleyj uucp 628766 May 11 08:41 sub-patid_dir-PA_run-02_epi.nii.gz -rw-r--r--+ 1 kenleyj uucp 628950 May 11 08:41 sub-patid_dir-AP_run-02_epi.nii.gz -rw-r--r--+ 1 kenleyj uucp 3178 May 11 08:41 sub-patid_dir-PA_run-01_epi.json -rw-r--r--+ 1 kenleyj uucp 3172 May 11 08:41 sub-patid_dir-AP_run-01_epi.json -rw-r--r--+ 1 kenleyj uucp 3173 May 11 08:41 sub-patid_dir-PA_run-02_epi.json -rw-r--r--+ 1 kenleyj uucp 3178 May 11 08:41 sub-patid_dir-AP_run-02_epi.json

patid/sub-patid/anat: total 24718 -rw-r--r--+ 1 kenleyj uucp 14178622 May 11 08:39 sub-patid_T1w.nii.gz -rw-r--r--+ 1 kenleyj uucp 11038105 May 11 08:41 sub-patid_T2w.nii.gz -rw-r--r--+ 1 kenleyj uucp 1969 May 11 08:41 sub-patid_T1w.json -rw-r--r--+ 1 kenleyj uucp 2000 May 11 08:41 sub-patid_T2w.json

arueter1 commented 2 years ago

@jkkenley - Were you able to send this over to Thomas? If you no longer need assistance, feel free to close this issue. :)

jkkenley commented 2 years ago

@Amanda @.***>/Thomas

I tried sending the cleaning file several times, but it kept getting kicked back. Trying a different approach now. Thomas, thank you for your help.

Jeanette

The materials in this email are private and may contain Protected Health Information. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying, distribution or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return email.

From: Amanda Rueter @.> Sent: Wednesday, June 22, 2022 3:25 PM To: DCAN-Labs/abcd-hcp-pipeline @.> Cc: Kenley, Jeanette @.>; Mention @.> Subject: Re: [DCAN-Labs/abcd-hcp-pipeline] dataset_description.json (Issue #79)

@jkkenleyhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fjkkenley&data=05%7C01%7Cjkkenley%40wustl.edu%7C24ddb15c26564c8b9c8f08da548d66df%7C4ccca3b571cd4e6d974b4d9beb96c6d6%7C0%7C0%7C637915263540539252%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=vPPb8BM0%2FwgVuxo7B1UQLf7kaA6vZfJv7eWWrKryohI%3D&reserved=0 - Were you able to send this over to Thomas? If you no longer need assistance, feel free to close this issue. :)

— Reply to this email directly, view it on GitHubhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FDCAN-Labs%2Fabcd-hcp-pipeline%2Fissues%2F79%23issuecomment-1163566619&data=05%7C01%7Cjkkenley%40wustl.edu%7C24ddb15c26564c8b9c8f08da548d66df%7C4ccca3b571cd4e6d974b4d9beb96c6d6%7C0%7C0%7C637915263540539252%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=vk7I4Vx4dTJMCfXfBrUGcyHR%2FyeG9Dx4GeQ4mk9kukQ%3D&reserved=0, or unsubscribehttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAUYL7WSW5SEWH6KYYMVQ6CDVQNZE3ANCNFSM5VVKRNEA&data=05%7C01%7Cjkkenley%40wustl.edu%7C24ddb15c26564c8b9c8f08da548d66df%7C4ccca3b571cd4e6d974b4d9beb96c6d6%7C0%7C0%7C637915263540539252%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Y6ft13PzXfk2TPW2qG1%2FuSMkZKxEAnNXYb4EkULFEzM%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.***>

The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

jarodroland commented 1 year ago

I came across this thread while searching a similar error message. For my data, it turned out that one of my field map json files was missing the IntendedFor entry. I was confused because when I compared my AP fmap json to a known good subject everything looked fine (similar to the above user), but my PA fmap was missing the IntendedFor entry. Once I corrected that it (so both the AP and PA fmap json had the correct IntendedFor directive) then it resolved the error. Hope this is helpful.

jkkenley commented 1 year ago

Thanks so much! I'll give this a try!

The materials in this email are private and may contain Protected Health Information. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying, distribution or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return email.

From: Jarod @.> Sent: Wednesday, September 14, 2022 5:57 PM To: DCAN-Labs/abcd-hcp-pipeline @.> Cc: Kenley, Jeanette @.>; Mention @.> Subject: Re: [DCAN-Labs/abcd-hcp-pipeline] dataset_description.json (Issue #79)

I came across this thread while searching a similar error message. For my data, it turned out that one of my field map json files was missing the IntendedFor entry. I was confused because when I compared my AP fmap json to a known good subject everything looked fine (similar to the above user), but my PA fmap was missing the IntendedFor entry. Once I corrected that it (so both the AP and PA fmap json had the correct IntendedFor directive) then it resolved the error. Hope this is helpful.

— Reply to this email directly, view it on GitHubhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FDCAN-Labs%2Fabcd-hcp-pipeline%2Fissues%2F79%23issuecomment-1247382706&data=05%7C01%7Cjkkenley%40wustl.edu%7Cce35f62c582441fd21df08da96a48356%7C4ccca3b571cd4e6d974b4d9beb96c6d6%7C0%7C0%7C637987930570302957%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=zt86symem2LGPMEwPwNN9BF0zSzihFqZk4djeX7ECuI%3D&reserved=0, or unsubscribehttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAUYL7WVJ62RJTHE555QEU6DV6JJ5ZANCNFSM5VVKRNEA&data=05%7C01%7Cjkkenley%40wustl.edu%7Cce35f62c582441fd21df08da96a48356%7C4ccca3b571cd4e6d974b4d9beb96c6d6%7C0%7C0%7C637987930570302957%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=CLOQkYYlrKhJftQb3lF216yybLnW2QNtb3S59aWWmZc%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.***>

The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

jkkenley commented 1 year ago

Any further updates on this by chance?

The materials in this email are private and may contain Protected Health Information. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying, distribution or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return email.

From: Kenley, Jeanette @.> Sent: Friday, May 13, 2022 8:28 AM To: DCAN-Labs/abcd-hcp-pipeline @.>; DCAN-Labs/abcd-hcp-pipeline @.> Cc: Author @.> Subject: Re: [DCAN-Labs/abcd-hcp-pipeline] dataset_description.json (Issue #79)

1.) "BandwidthPerPixelPhaseEncode": 21.786, "EffectiveEchoSpacing": 0.000510012, "DerivedVendorReportedEchoSpacing": 0.000510012, "TotalReadoutTime": 0.045391,

2.) "InPlanePhaseEncodingDirectionDICOM": "COL", "ConversionSoftware": "dcm2niix", "ConversionSoftwareVersion": "v1.0.20210317", "Dcm2bidsVersion": "2.1.6"

These parameters are consistet with a subject who did run.

Thanks, Jeanette

The materials in this email are private and may contain Protected Health Information. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying, distribution or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return email.

From: Thomas Madison @.> Sent: Thursday, May 12, 2022 6:02 PM To: DCAN-Labs/abcd-hcp-pipeline @.> Cc: Kenley, Jeanette @.>; Author @.> Subject: Re: [DCAN-Labs/abcd-hcp-pipeline] dataset_description.json (Issue #79)

Hi Jeanette,

My best guess is the error is unrelated to dataset_description.json and the actual problem is the "EffectiveEchoSpacing" field is missing from the fmap json files. (I believe older versions of Dcm2bids do not output EffectiveEchoSpacing, so that may be the cause.) Can you check:

  1. If "EffectiveEchoSpacing" is present in the fmap directory json files
  2. If the json files have infomation about the Dcm2bids or dcm2niix version used to convert from DICOM (for Dcm2bids the field is titled "Dcm2bidsVersion")

Thanks!

— Reply to this email directly, view it on GitHubhttps://github.com/DCAN-Labs/abcd-hcp-pipeline/issues/79#issuecomment-1125493627, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AUYL7WTMY6AHBWT4YRT3M7TVJWE2HANCNFSM5VVKRNEA. You are receiving this because you authored the thread.Message ID: @.***>

The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

lundq163 commented 1 year ago

Hi @jkkenley, were you able to successfully provide @madisoth with what he was looking for? (the BIDS directory tree for the subject, the command or script used to run the pipeline)

fpichardo-umn commented 1 year ago

I am having the same issue described above except that I'm testing running the pipeline on two participants, and it happens for both. Here's the directory structure: sub-001 sub-001/ses-06 sub-001/ses-06/anat sub-001/ses-06/anat/sub-001_ses-06_T1w.json sub-001/ses-06/anat/sub-001_ses-06_T1w.nii.gz sub-001/ses-06/dwi sub-001/ses-06/dwi/sub-001_ses-06_dir-PA_dwi.bval sub-001/ses-06/dwi/sub-001_ses-06_dir-PA_dwi.json sub-001/ses-06/dwi/sub-001_ses-06_dir-AP_dwi.bval sub-001/ses-06/dwi/sub-001_ses-06_dir-AP_dwi.bvec sub-001/ses-06/dwi/sub-001_ses-06_dir-AP_dwi.json sub-001/ses-06/dwi/sub-001_ses-06_dir-PA_dwi.bvec sub-001/ses-06/dwi/sub-001_ses-06_dir-PA_dwi.nii.gz sub-001/ses-06/dwi/sub-001_ses-06_dir-AP_dwi.nii.gz sub-001/ses-06/fmap sub-001/ses-06/fmap/sub-001_ses-06_acq-rest_dir-AP_epi.json sub-001/ses-06/fmap/sub-001_ses-06_acq-hariri_dir-AP_epi.json sub-001/ses-06/fmap/sub-001_ses-06_acq-hariri_dir-AP_epi.nii.gz sub-001/ses-06/fmap/sub-001_ses-06_acq-rest_dir-AP_epi.nii.gz sub-001/ses-06/func sub-001/ses-06/func/sub-001_ses-06_task-rest_bold.nii.gz sub-001/ses-06/func/sub-001_ses-06_task-rest_bold.json sub-001/ses-06/func/sub-001_ses-06_task-hariri_bold.nii.gz sub-001/ses-06/func/sub-001_ses-06_task-hariri_bold.json sub-001/ses-06/func/sub-001_ses-06_task-hariri_events.tsv sub-001/ses-06/sub-001_ses-06_scans.json sub-001/ses-06/sub-001_ses-06_scans.tsv

EffectiveEchoSpacing is in the fmap jsons, and they all have IntendedFor in the relevant fmap json.

I've run it via as an sbatch script that runs the following command: env -i singularity run \ -B /PATH/TO/DATA:/bids_input \ -B /PATH/TO/OUTPUT:/output \ -B /PATH/TO/license.txt:/opt/freesurfer/license.txt \ ./abcd-hcp-pipeline.sif /bids_input /output --freesurfer-license=/opt/freesurfer/license.txt --participant-label 001 --session-id 06

Using version: abcd-hcp-pipeline 0.1.2

Here are the resources I requested when submitting the job:

SBATCH --time=48:00:00

SBATCH --ntasks=8

SBATCH --mem=32g

SBATCH --tmp=32g

I also loaded the following modules: module load fsl module load python3 module load freesurfer module load singularity

Appreciate the help!

madisoth commented 1 year ago

@fpichardo-umn

The pipeline will assume that since your fmap filenames end in "_epi.nii.gz", you have PEPolar fmaps (e.g. both "dir-AP" and "dir-PA") for topup correction. With only "dir-AP" in the fmap dir an error would be expected.

The fieldmap schemes the pipelines can handle are cases 2 and 4 in the BIDS spec here: https://bids-specification.readthedocs.io/en/stable/04-modality-specific-files/01-magnetic-resonance-imaging-data.html#types-of-fieldmaps

Another potential issue is that the BIDS "acq-" entity in filenames may go unrecognized (it's a known issue with the pipeline), so you may need to rename files from this: 


 sub-001/ses-06/fmap/sub-001_ses-06_acq-hariri_dir-AP_epi.nii.gz
 sub-001/ses-06/fmap/sub-001_ses-06_acq-rest_dir-AP_epi.nii.gz

to this:

 sub-001/ses-06/fmap/sub-001_ses-06_dir-AP_run-01_epi.nii.gz
 sub-001/ses-06/fmap/sub-001_ses-06_dir-AP_run-02_epi.nii.gz

Also, if running on the MSI HPC system via Singularity container then Singularity should be the only module you need to load, all other dependencies (other than the FreeSurfer license) are built into the container.

fpichardo-umn commented 1 year ago

This fixed the issue (specifically checking on the fieldmaps and making sure there were two there). Thanks for the help!