search

PennLINC / xcpEngine

Official public repository for the XCP Engine. This tool is deprecated in favor of XCP-D and ASLPrep.
MIT License
66 stars 42 forks source link

Editing design file fails to produce a confound file #426

Closed roeysc closed 3 years ago

roeysc commented 3 years ago

Describe the bug I have data preprocessed with fMRIPrep v20.2.1. I tried running xcpengine with the acompcor pipeline, with additional global signal regression. For that, I made a copy of the fc-acompcor.dsn file, and changed “confound2_gsr[2]=0” to “confound2_gsr[2]=1”. Looking at the “confound2” output directory, I see that a file names “sub-003_task_rest_run-1_gsr.1D” was not created.

Cohort file Paste cohort file between the triple backticks

id0, id1,img
sub-003,task-rest_run-1,sub-003/func/sub-003_task-rest_run-1_space-T1w_desc-preproc_bold.nii.gz
sub-003,task-rest_run-2,sub-003/func/sub-003_task-rest_run-2_space-T1w_desc-preproc_bold.nii.gz

Design File Paste your entire design (.dsn) file between the triple backticks

#!/usr/bin/env bash

###################################################################
#  \u2297  \u2297  \u2297  \u2297  \u2297  \u2297  \u2297  \u2297  \u2297  \u2297  \u2297  \u2297  \u2297  \u2297  \u2297  \u2297  \u2297  \u2297  \u2297  \u2297  \u2297  #
###################################################################

###################################################################
# This design file stores the values of all variables required to
# execute a complete neuroimage processing pipeline. You may
# execute the analysis specified in this design file by calling
# (in any v4 or higher bash terminal):
#
# xcpEngine -d <design> -c <cohort> -o <output> <options>
#
# Variables fall into five general categories:
# * ANALYSIS VARIABLES are used at all stages of this analysis.
# * PIPELINE specifies the modules that comprise the analysis.
# * MODULE VARIABLES are used during one stage of the analysis.
#                  These are typically array variables with array
#                  indices equal to the index of the module that
#                  calls them.
# * OUTPUT VARIABLES may be used at all stages of the analysis.
#                  These are typically array variables with array
#                  indices equal to the value of the primary
#                  subject identifier. They will appear only in
#                  localised design files.
###################################################################

###################################################################
# ANALYSIS VARIABLES
###################################################################

analysis=fc_$(whoami)
design=/ems/elsc-labs/mezer-a/code/roey/HartLab/xcpengine/fc-acompcor_gsr.dsn
sequence=anatomical
standard=MNI%2x2x2

###################################################################
# PIPELINE
###################################################################

pipeline=prestats,confound2,regress,fcon,reho,alff,roiquant,norm,qcfc

###################################################################
# 1 PRESTATS
###################################################################

prestats_rerun[1]=1
prestats_cleanup[1]=1
prestats_process[1]=FMP

###################################################################
# 2 CONFOUND2
###################################################################

confound2_rps[2]=1
confound2_rms[2]=0
confound2_wm[2]=0
confound2_csf[2]=0
confound2_gsr[2]=1
confound2_acompcor[2]=1
confound2_tcompcor[2]=0
confound2_aroma[2]=0
confound2_past[2]=0
confound2_dx[2]=1
confound2_sq[2]=1
confound2_censor[2]=0
confound2_censor_contig[2]=0
confound2_rerun[2]=1
confound2_cleanup[2]=1

###################################################################
# 3  REGRESS
###################################################################

regress_tmpf[3]=butterworth
regress_hipass[3]=0.01
regress_lopass[3]=0.08
regress_tmpf_order[3]=1
regress_tmpf_pass[3]=2
regress_tmpf_ripple[3]=0.5
regress_tmpf_ripple2[3]=20
regress_dmdt[3]=2
regress_1ddt[3]=1
regress_smo[3]=6
regress_sptf[3]=susan
regress_usan[3]=default
regress_usan_space[3]=
regress_rerun[3]=0
regress_cleanup[3]=1
regress_process[3]=DMT-TMP-REG

###################################################################
# 4 FCON
###################################################################

fcon_atlas[4]=power264
fcon_metric[4]=corrcoef
fcon_rerun[4]=0
fcon_cleanup[4]=1

###################################################################
# 5 REHO
###################################################################

reho_nhood[5]=vertices
reho_roikw[5]=0 # does nothing at the moment
reho_sptf[5]=susan
reho_smo[5]=6
reho_rerun[5]=0
reho_cleanup[5]=1

###################################################################
# 6 ALFF
###################################################################

alff_hipass[6]=0.01
alff_lopass[6]=0.08
alff_sptf[6]=susan
alff_smo[6]=6
alff_rerun[6]=0
alff_cleanup[6]=1

###################################################################
# 7 ROIQUANT
###################################################################

roiquant_atlas[7]=power264
roiquant_globals[7]=1
roiquant_vol[7]=0
roiquant_rerun[7]=0
roiquant_cleanup[7]=1

###################################################################
# 8 NORM
###################################################################
norm_primary[8]=1
norm_rerun[8]=0
norm_cleanup[8]=1

##################################################################
# 9 QCFC
###################################################################
qcfc_atlas[9]=power264
qcfc_sig[9]=fdr
qcfc_rerun[9]=0
qcfc_cleanup[9]=1

Error message

No error message, but it seems to fail producing the required file.

Runtime Information I used Docker to run xcpengine.

I also asked about this in neurostars.

Many thanks for your help!

a3sha2 commented 3 years ago

can you post the command line ? thanks

roeysc commented 3 years ago

Of course. Thank you, @a3sha2.

xcpengine-docker -c /ems/elsc-labs/mezer-a/code/roey/HartLab/xcpengine/cohort_sub_003.csv -d /ems/elsc-labs/mezer-a/code/roey/HartLab/xcpengine/fc-acompcor_gsr.dsn -i /tmp/xcpengingTmp -o /ems/elsc-labs/mezer-a/Mezer-Lab/rawData/HUJI/Collaboration_YuvalHart/BIDS/derivatives/xcpengine/fc-acompcor_gsr/sub-003 -r /ems/elsc-labs/mezer-a/Mezer-Lab/rawData/HUJI/Collaboration_YuvalHart/BIDS/derivatives/fmriprep

roeysc commented 3 years ago

Is there any other information I could provide that might help? Many thanks, Roey

roeysc commented 3 years ago

Dear all,

I am still unable to resolve this issue. I will provide some more details here. This might be a very early problem, as the log file reads already in the PRESTATS module:

>> 3dinfo -orient
[7m** ERROR:[0m No dsets on command line? I have nothing to do.

Most output directories are empty: The prestats directory only has an empty "mc" folder. The confound2 directory only includes the file "sub-003_task-rest_run-1_modelParameterCount.txt" and the motion correction folder ("mc") with 8 files in it.

The design and cohort files are in the same directory, and have rwx permissions.

I would be grateful for any help. Best, Roey

a3sha2 commented 3 years ago

I am sorry for late response, it look like input data error.design file is good. Please can you try docker or singularity image. that suppose to work apology pls Roey

roeysc commented 3 years ago

Hi Azeez, Thanks for getting back to me! I was using the xcpengine-docker. Should using singularity instead of docker make a difference? Thanks! Roey

a3sha2 commented 3 years ago

Yes, pls use singularity image https://xcpengine.readthedocs.io/containers/index.html

roeysc commented 3 years ago

Dear Azeez, I have built the Singularity image. I ran it like so:

singularity run -B /ems/elsc-labs/mezer-a/Mezer-Lab/rawData/HUJI/Collaboration_YuvalHart/BIDS/derivatives/fmriprep:/data xcpEngine.simg -c /data/xcpengine_cohort_sub_016.csv -d /data/fc-acompcor-gsr.dsn -o /data/derivatives/xcpengine/fc-acompcor_gsr_Singularity -i /tmp

With the following cohort file:

id0,id1,img
sub-016,task-rest_run-1,/data/sub-016/func/sub-016_task-rest_run-1_space-T1w_desc-preproc_bold.nii.gz
sub-016,task-rest_run-2,/data/sub-016/func/sub-016_task-rest_run-2_space-T1w_desc-preproc_bold.nii.gz
sub-016,task-cfg_run-1,/data/sub-016/func/sub-016_task-cfg_run-1_space-T1w_desc-preproc_bold.nii.gz
sub-016,task-motor_run-1,/data/sub-016/func/sub-016_task-motor_run-1_space-T1w_desc-preproc_bold.nii.gz

But I got this error: The primary analyte is not a valid image: /data/sub-016/func/sub-016_task-motor_run-1_space-T1w_desc-preproc_bold.nii.gz

I made sure to bind the fmriprep output folder (following issue #427). I also made sure that the design file has read and write permission.

I also tried using the -r flag:

singularity run -B /ems/elsc-labs/mezer-a/Mezer-Lab/rawData/HUJI/Collaboration_YuvalHart/BIDS/derivatives/fmriprep:/data xcpEngine.simg -c /data/xcpengine_cohort_sub_016.csv -d /data/fc-acompcor-gsr.dsn -o /data/derivatives/xcpengine/fc-acompcor_gsr_Singularity -r /ems/elsc-labs/mezer-a/Mezer-Lab/rawData/HUJI/Collaboration_YuvalHart/BIDS/derivatives/fmriprep -i /tmp

With this cohort file:

id0,id1,img
sub-016,task-rest_run-1,/sub-016/func/sub-016_task-rest_run-1_space-T1w_desc-preproc_bold.nii.gz
sub-016,task-rest_run-2,/sub-016/func/sub-016_task-rest_run-2_space-T1w_desc-preproc_bold.nii.gz
sub-016,task-cfg_run-1,/sub-016/func/sub-016_task-cfg_run-1_space-T1w_desc-preproc_bold.nii.gz
sub-016,task-motor_run-1,/sub-016/func/sub-016_task-motor_run-1_space-T1w_desc-preproc_bold.nii.gz

But that didn't work either.

Edit: Perhaps the problem is that the imtest function of FSL is not recognized by the xcpengine container of Singularity?

· [Verifying that images are present]
/xcpEngine/core/functions/is_image: line 2: /ems/elsc-labs/mezer-a/fsl/bin/imtest: No such file or directory
/xcpEngine/core/functions/is_image: line 2: ((: == 1 : syntax error: operand expected (error token is "== 1 ")

· [A major error has occurred.]
· [The processing stream will now abort.]

Any help would be greatly appreciated! Roey

a3sha2 commented 3 years ago

yes, the -r flag is not configure very well it should start with -r /data/ not -r /ems/elsc-lab .. @mattcieslak

roeysc commented 3 years ago

Thanks, Azeez! I'm afraid that didn't work either. The command:

singularity run -B /ems/elsc-labs/mezer-a/Mezer-Lab/rawData/HUJI/Collaboration_YuvalHart/BIDS/derivatives/fmriprep:/data xcpEngine.simg -c /data/xcpengine_cohort_sub_016.csv -d /data/fc-acompcor-gsr.dsn -o /data/derivatives/xcpengine/fc-acompcor_gsr_Singularity -r /data/ -i /tmp

Resulted in the same error.

a3sha2 commented 3 years ago

do you have fmriprep output in this directory /ems/elsc-labs/mezer-a/Mezer-Lab/rawData/HUJI/Collaboration_YuvalHart/BIDS/derivatives/fmriprep pls can you share the full log

roeysc commented 3 years ago

Yes, that's the output directory of fmriprep (it include the sub-folder sub-001, sub-002 etc.).

In addition, I should note that in the output directory of xcpengine, it also created a directory named preproc_bold.nii.gz, as this were another subject. So the output directory includes the following folder: group ; preproc_bold.nii.gz ; sub-016.

I couldn't find the correct log file, so I'm pasting the output form the terminal. Would that suffice? Thank you, Azeez! Roey

Received options: -c /data/xcpengine_cohort_sub_016.csv -d /data/fc-acompcor-gsr.dsn -o /data/derivatives/xcpengine/fc-acompcor_gsr_Singularity -r /data -i /tmp
$XCPEDIR is /xcpEngine

Constructing a pipeline based on user specifications
····································································
· [D][/data/fc-acompcor-gsr.dsn]
· [I][/tmp]
· [C][/data/xcpengine_cohort_sub_016.csv]
· [O][/data/derivatives/xcpengine/fc-acompcor_gsr_Singularity]
· [M][Executing serially]
····································································
Date: 202105070518
UUID: d36b6f5f-4620-4ef8-8c1a-ba7cd6d6071c

Checking general dependencies
····································································
· Version AFNI           AFNI_21.1.01 
· Version ANTs           3.0.0.0 
· [Warning: FSL version unknown]
· Version C3D            1.1.0 
· Version JQ             jq-1.5
· Version XCP Engine     xcpEngine-v1.2.3
· R version              3.3.3 
· R scripting front-end  3.3.3 
· · RNifti version       1.3.0
· · optparse version     1.6.6
· · pracma version       2.3.3
· · signal version       0.7.6
····································································
All general dependencies are present.

Checking environment
All environmental variables are defined.

Checking module-specific dependencies

###################################################################
#                                            #
#                                                                 #
#                        XCP \u2605 \u2605 \u2605 ENGINE                       #
#                                                                 #
#                                            #
###################################################################

###################################################################
#                                            #
#                                                                 #
                  CURRENTLY PROCESSING MODULE:
                         \u2605 LOCALISER \u2605
#                                                                 #
#                                            #
###################################################################

[Delegating control to module]

###################################################################
#                                            #
#                                                                 #
#                     XCP ENGINE LOCALISER                      #
#                                                                 #
#                                            #
###################################################################

Current processing step:
Localising subject metadata...
····································································
· [Parsing subject identifiers]
· [Creating output directory]
· [/data/derivatives/xcpengine/fc-acompcor_gsr_Singularity/sub-016/task-rest_run-1]
· [Initialising space mapping]
· [Initialising derivative maps]
· [Initialising atlas system]
· Ingress
· [Native space: sub-016_task-rest_run-1_anatomical]
· [Standard space: MNI]
· [Anatomical space: sub-016_task-rest_run-1_anatomical]
/xcpEngine/core/functions/is_image: line 2: /ems/elsc-labs/mezer-a/fsl/bin/imtest: No such file or directory
/xcpEngine/core/functions/is_image: line 2: ((: == 1 : syntax error: operand expected (error token is "== 1 ")
fixed '/xcpEngine/space/MNI/MNI-2x2x2.nii.gz' is not an image
· [Priming analytic audit]
· [Priming quality assessment]
····································································
Processing step complete:
Localising subject metadata...

Current processing step:
Localising image
····································································
· [Verifying that images are present]
/xcpEngine/core/functions/is_image: line 2: /ems/elsc-labs/mezer-a/fsl/bin/imtest: No such file or directory
/xcpEngine/core/functions/is_image: line 2: ((: == 1 : syntax error: operand expected (error token is "== 1 ")

· [A major error has occurred.]
· [The processing stream will now abort.]
· [Preparing diagnostics]
····································································

Module Workflow Map
····································································

· @1
· @1.1
· @1.1a
· @1.1b
· @1.2
· @1.3
· @1.4
· @1.5
· @1.5.1
· @1.5.2a
· @1.5.3a
· @1.5
· @1.6
· @2
· @2.1
· ERROR
····································································

····································································
· [An error occurred while processing module localiser.]
· [The error was detected at signpost @2.1.]
· [The most recent command logged was]
· []
· [For additional details, view the processing log. For improved]
· [diagnostics, increment verbosity using the -t option.]
· []
· [The primary analyte is not a valid image: /data/sub-016/func/sub-016_task-rest_run-1_space-T1w_desc-preproc_bold.nii.gz]
· []
· [stream abort]
····································································

###################################################################
#                                            #
#                                                                 #
#                     XCP ENGINE LOCALISER                      #
#                                                                 #
#                                            #
###################################################################

Current processing step:
Localising subject metadata...
····································································
· [Parsing subject identifiers]
· [Creating output directory]
· [/data/derivatives/xcpengine/fc-acompcor_gsr_Singularity/preproc_bold.nii.gz]
· [Initialising space mapping]
· [Initialising derivative maps]
· [Initialising atlas system]
· Ingress
· [Native space: preproc_bold.nii.gz_anatomical]
· [Standard space: MNI]
· [Anatomical space: preproc_bold.nii.gz_anatomical]
/xcpEngine/core/functions/is_image: line 2: /ems/elsc-labs/mezer-a/fsl/bin/imtest: No such file or directory
/xcpEngine/core/functions/is_image: line 2: ((: == 1 : syntax error: operand expected (error token is "== 1 ")
fixed '/xcpEngine/space/MNI/MNI-2x2x2.nii.gz' is not an image
· [Priming analytic audit]
· [Priming quality assessment]
····································································
Processing step complete:
Localising subject metadata...

Current processing step:
Localising image
····································································
· [Verifying that images are present]
/xcpEngine/core/functions/is_image: line 2: /ems/elsc-labs/mezer-a/fsl/bin/imtest: No such file or directory
/xcpEngine/core/functions/is_image: line 2: ((: == 1 : syntax error: operand expected (error token is "== 1 ")

· [A major error has occurred.]
· [The processing stream will now abort.]
· [Preparing diagnostics]
····································································

Module Workflow Map
····································································

· @1
· @1.1
· @1.1a
· @1.1b
· @1.2
· @1.3
· @1.4
· @1.5
· @1.5.1
· @1.5.2a
· @1.5.3a
· @1.5
· @1.6
· @2
· @2.1
· ERROR
····································································

····································································
· [An error occurred while processing module localiser.]
· [The error was detected at signpost @2.1.]
· [The most recent command logged was]
· []
· [For additional details, view the processing log. For improved]
· [diagnostics, increment verbosity using the -t option.]
· []
· [The primary analyte is not a valid image: /data]
· []
· [stream abort]
····································································

###################################################################
#                                            #
#                                                                 #
                  CURRENTLY PROCESSING MODULE:
                         \u2605 prestats \u2605
#                                                                 #
#                                            #
###################################################################

[Delegating control to module]

····································································
· [Module failure occurred in a number of subjects. They have been]
· [removed from further analysis.]
· · [sub-016_task-rest_run-1 removed from analysis]
· · [preproc_bold.nii.gz removed from analysis]
····································································

###################################################################
#                                            #
#                                                                 #
                  CURRENTLY PROCESSING MODULE:
                         \u2605 confound2 \u2605
#                                                                 #
#                                            #
###################################################################

[Delegating control to module]

····································································
· [Module failure occurred in a number of subjects. They have been]
· [removed from further analysis.]
· · [sub-016_task-rest_run-1 removed from analysis]
· · [preproc_bold.nii.gz removed from analysis]
····································································

###################################################################
#                                            #
#                                                                 #
                  CURRENTLY PROCESSING MODULE:
                         \u2605 regress \u2605
#                                                                 #
#                                            #
###################################################################

[Delegating control to module]

····································································
· [Module failure occurred in a number of subjects. They have been]
· [removed from further analysis.]
· · [sub-016_task-rest_run-1 removed from analysis]
· · [preproc_bold.nii.gz removed from analysis]
····································································

###################################################################
#                                            #
#                                                                 #
                  CURRENTLY PROCESSING MODULE:
                         \u2605 fcon \u2605
#                                                                 #
#                                            #
###################################################################

[Delegating control to module]

····································································
· [Module failure occurred in a number of subjects. They have been]
· [removed from further analysis.]
· · [sub-016_task-rest_run-1 removed from analysis]
· · [preproc_bold.nii.gz removed from analysis]
····································································

###################################################################
#                                            #
#                                                                 #
                  CURRENTLY PROCESSING MODULE:
                         \u2605 reho \u2605
#                                                                 #
#                                            #
###################################################################

[Delegating control to module]

····································································
· [Module failure occurred in a number of subjects. They have been]
· [removed from further analysis.]
· · [sub-016_task-rest_run-1 removed from analysis]
· · [preproc_bold.nii.gz removed from analysis]
····································································

###################################################################
#                                            #
#                                                                 #
                  CURRENTLY PROCESSING MODULE:
                         \u2605 alff \u2605
#                                                                 #
#                                            #
###################################################################

[Delegating control to module]

····································································
· [Module failure occurred in a number of subjects. They have been]
· [removed from further analysis.]
· · [sub-016_task-rest_run-1 removed from analysis]
· · [preproc_bold.nii.gz removed from analysis]
····································································

###################################################################
#                                            #
#                                                                 #
                  CURRENTLY PROCESSING MODULE:
                         \u2605 roiquant \u2605
#                                                                 #
#                                            #
###################################################################

[Delegating control to module]

····································································
· [Module failure occurred in a number of subjects. They have been]
· [removed from further analysis.]
· · [sub-016_task-rest_run-1 removed from analysis]
· · [preproc_bold.nii.gz removed from analysis]
····································································

###################################################################
#                                            #
#                                                                 #
                  CURRENTLY PROCESSING MODULE:
                         \u2605 norm \u2605
#                                                                 #
#                                            #
###################################################################

[Delegating control to module]

····································································
· [Module failure occurred in a number of subjects. They have been]
· [removed from further analysis.]
· · [sub-016_task-rest_run-1 removed from analysis]
· · [preproc_bold.nii.gz removed from analysis]
····································································

###################################################################
#                                            #
#                                                                 #
                  CURRENTLY PROCESSING MODULE:
                         \u2605 qcfc \u2605
#                                                                 #
#                                            #
###################################################################

[Delegating control to module]

····································································
· [Module failure occurred in a number of subjects. They have been]
· [removed from further analysis.]
· · [sub-016_task-rest_run-1 removed from analysis]
· · [preproc_bold.nii.gz removed from analysis]
····································································
generating report

###################################################################
#                                            #
#                                                                 #
#                       ANALYSIS COMPLETE                       #
#                                                                 #
#                                            #
###################################################################
roeysc commented 3 years ago

Hi Azeez, Is this enough information to see what the source of the error might be? Thank you! Roey

roeysc commented 3 years ago

Hi Azeez and everyone, Following you advice, to avoid the docker imaging interacting with the local environment I have added the flag --cleanenv:

singularity run --cleanenv -B ….

This seems to work perfectly. It is my understanding that I should run singularity directly, and not with xcpengine-singularity.

Thanks a lot, Azeez!