DCANBOLDProcessing Step Error (possibly file naming issue?)

iamdamion commented 2 years ago

First, I am using the current docker image of the pipeline pulled from your docker hub about 4 weeks ago. (The -v says "abcd-hcp-pipeline 0.0.1" but I think the file just wasn't updated. According to the docker hub I believe it's version 0.0.3?)

I have tested and QA'd outputs all steps up to the DCANBOLD step. for the DCANBOLD step, I believe a file naming issue is creating a series of file not found errors. Here is an example of the error (this is repeated for all functional/task scans) from the DCANBOLDProcessing_teardown.err log):

While running:
/opt/workbench/bin_linux64/../exe_linux64/wb_command -cifti-parcellate /output/sub-MSCPI13/ses-20190223/files/MNINonLinear/Results/task-rest_DCANBOLDProc_v4.0.0_Atlas.dtseries.nii /opt/dcan-tools/dcan_bold_proc/templates/parcellations/Gordon/fsLR/Gordon.32k_fs_LR.dlabel.nii COLUMN /output/sub-MSCPI13/ses-20190223/files/MNINonLinear/Results/task-rest_DCANBOLDProc_v4.0.0_Gordon.ptseries.nii

ERROR: failed to open file '/output/sub-MSCPI13/ses-20190223/files/MNINonLinear/Results/task-rest_DCANBOLDProc_v4.0.0_Atlas.dtseries.nii', file does not exist, or folder permissions prevent seeing it

Within that files/MNINonLinear/Results/ folder, I have one dtseries for each task, however, they are named with an extra zero as such:

task-rest0_DCANBOLDProc_v4.0.0_Atlas.dtseries.nii

I believe that zero in the file name is why this step is failing, but am unsure why it's being written with that zero. If someone could look into this I'd appreciate it. Please also let me know if I should upload any other logs or info to help identify what's happening.

I think I'm left with fully pre-processed and concatenated dtseries files....but all the timeseries extraction, etc steps are failing. My summary folder has all the motion files, but the analyses_V2 subfolders are all empty.

madisoth commented 2 years ago

Hi Damion,

The short explanation is that the regexes in the pipeline wrapper to extract BIDS fields don't match the regexes in the included version of DCANBOLDProcessing, which is causing your filename problem.

Issue on the dcan_bold_processing repo with details: https://github.com/DCAN-Labs/dcan_bold_processing/issues/5
PR to resolve the issue: https://github.com/DCAN-Labs/dcan_bold_processing/pull/15
PR to update abcd-hcp-pipeline with the fixed dcan_bold_processing is here: https://github.com/DCAN-Labs/abcd-hcp-pipeline/pull/39 . Still needs to be reviewed and merged, so for now you can get it on my personal Dockerhub (tag 10222021a): https://hub.docker.com/r/madisoth/abcd-hcp-pipeline/tags

arueter1 commented 2 years ago

Thanks for your help with this @madisoth ! I'm going to mark this as closed as we have a process in place to fix the issue as Thomas outlined above. If you have further questions, please feel free to reopen it, @iamdamion !

iamdamion commented 2 years ago

@madisoth Finally got my job to run, and it looks like there's some typos in the version I pulled from your docker hub. I'm still stuck trying to get through the dcan bold processing step. I'm not sure what version of the python script is in the docker I pulled, so I haven't checked this standard out variable. But here is the error from my teardown file. Please let me know if you need other log info to help track this down.

ERROR: Could not open file
Traceback (most recent call last):
  File "/opt/dcan-tools/dcan_bold_proc/dcan_bold_proc.py", line 812, in <module>
    _cli()
  File "/opt/dcan-tools/dcan_bold_proc/dcan_bold_proc.py", line 78, in _cli
    return interface(**kwargs)
  File "/opt/dcan-tools/dcan_bold_proc/dcan_bold_proc.py", line 388, in interface
    repetition_time = get_repetition_time(input_spec['fmri_volume'])
  File "/opt/dcan-tools/dcan_bold_proc/dcan_bold_proc.py", line 578, in get_repetition_time
    repetition_time = float(stdout)
ValueError: could not convert string to float:

@arueter1 I am unable to reopen this issue since I didn't close it. Could you, please? Thanks.

madisoth commented 2 years ago

Hi Damion,

Can you upload or email me (tmadison [at] umn [dot] edu) the full logs directory for the session?

It looks like an expected file <sub>/<ses>/files/MNINonLinear/Results/<ses_task_run>/<ses_task_run>.nii.gz is either missing, misnamed or not readable so I would also check if anything there looks wrong. Thanks!

iamdamion commented 2 years ago

Yeah, I'll email you all the logs now. If you think the above is because that standard out is just blank, I have noticed that the updated docker I pulled from your link is naming the folders and files with the session number. the old version did not do this. However, I thought that all these folders and files were made with the DCANBOLDProc step....but now that you mention this, maybe I need to go back to the Volume fmri step so the folders and files are made with the updated naming (session included)?

I'll email you my logs now though anyway, just in case.

iamdamion commented 2 years ago

This looks all good now. It was just an unexpected file naming issue since the new version adds session IDs to files and folders. After re-running volume and surface all finished without error. Thanks for the help and pointing me to the updated docker image.

DCAN-Labs / abcd-hcp-pipeline

DCANBOLDProcessing Step Error (possibly file naming issue?) #48