Error with regress module when running on hpc

[jasongong11]

Describe the bug The crash happened when running the regress module on singularity at hpc

Command I used

singularity run -B /group/rwhuskeygrp/bids_nii/derivatives/fmriprep-latest/fmriprep:/home/user/data \
/group/rwhuskeygrp/singularity_images/xcpEngine.simg \ 
-d /home/user/data/fc-aroma-gsr-06125.dsn \ 
-c /home/user/data/tempcohort/1.csv \ 
-o /home/user/data/output \ 
-r /home/user/data/ \ 
-i /tmp

Cohort file Paste cohort file between the triple backticks

id0,id1,img
sub-005,run-01,sub-005/func/sub-005_task-game_run-1_space-MNI152NLin2009cAsym_desc-preproc_bold.nii.gz
...,...,...

Design File Paste your entire design (.dsn) file between the triple backticks

#!/usr/bin/env bash

###################################################################
#  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  #
###################################################################

###################################################################
# This design file stores the values of all variables required to
# execute a complete neuroimage processing pipeline. You may
# execute the analysis specified in this design file by calling
# (in any v4 or higher bash terminal):
#
# xcpEngine -d <design> -c <cohort> -o <output> <options>
#
# Variables fall into five general categories:
# * ANALYSIS VARIABLES are used at all stages of this analysis.
# * PIPELINE specifies the modules that comprise the analysis.
# * MODULE VARIABLES are used during one stage of the analysis.
#                  These are typically array variables with array
#                  indices equal to the index of the module that
#                  calls them.
# * OUTPUT VARIABLES may be used at all stages of the analysis.
#                  These are typically array variables with array
#                  indices equal to the value of the primary
#                  subject identifier. They will appear only in
#                  localised design files.
###################################################################

###################################################################
# ANALYSIS VARIABLES
###################################################################

analysis=fc_$(whoami)
design=/home/user/data/fc-aroma-gsr-06125.dsn
sequence=anatomical
standard=MNI%2x2x2

###################################################################
# PIPELINE
###################################################################

pipeline=prestats,confound2,regress,fcon,reho,alff,roiquant,norm,qcfc

###################################################################
# 1 PRESTATS
###################################################################

prestats_rerun[1]=1
prestats_cleanup[1]=1
prestats_process[1]=FMP

###################################################################
# 2 CONFOUND2
###################################################################

confound2_rps[2]=0
confound2_rms[2]=0
confound2_wm[2]=1
confound2_csf[2]=1
confound2_gsr[2]=1
confound2_acompcor[2]=0
confound2_tcompcor[2]=0
confound2_aroma[2]=1
confound2_past[2]=0
confound2_dx[2]=0
confound2_sq[2]=1
confound2_censor[2]=0
confound2_censor_contig[2]=0
confound2_rerun[2]=0
confound2_cleanup[2]=1

###################################################################
# 3  REGRESS
###################################################################

regress_tmpf[3]=butterworth
regress_hipass[3]=0.06
regress_lopass[3]=0.125
regress_tmpf_order[3]=1
regress_tmpf_pass[3]=2
regress_tmpf_ripple[3]=0.5
regress_tmpf_ripple2[3]=20
regress_dmdt[3]=2
regress_1ddt[3]=1
regress_smo[3]=6
regress_sptf[3]=susan
regress_usan[3]=default
regress_usan_space[3]=
regress_rerun[3]=0
regress_cleanup[3]=1
regress_process[3]=DMT-TMP-REG

###################################################################
# 4 FCON
###################################################################

fcon_atlas[4]=power264
fcon_metric[4]=corrcoef
fcon_thr[4]=N
fcon_window[4]=10
fcon_pad[4]=FALSE
fcon_rerun[4]=0
fcon_cleanup[4]=1

###################################################################
# 5 REHO
###################################################################

reho_nhood[5]=vertices
reho_roikw[5]=0 # does nothing at the moment
reho_sptf[5]=susan
reho_smo[5]=6
reho_rerun[5]=0
reho_cleanup[5]=1

###################################################################
# 6 ALFF
###################################################################

alff_hipass[6]=0.06
alff_lopass[6]=0.125
alff_sptf[6]=susan
alff_smo[6]=6
alff_rerun[6]=0
alff_cleanup[6]=1

###################################################################
# 7 ROIQUANT
###################################################################

roiquant_atlas[7]=power264
roiquant_globals[7]=1
roiquant_vol[7]=0
roiquant_rerun[7]=0
roiquant_cleanup[7]=1

###################################################################
# 8 NORM
###################################################################
norm_primary[8]=1
norm_rerun[8]=0
norm_cleanup[8]=1

##################################################################
# 9 QCFC
###################################################################
qcfc_atlas[9]=power264
qcfc_sig[9]=fdr
qcfc_rerun[9]=0
qcfc_cleanup[9]=1

Error message Paste your error message between the backticks

###################################################################
#  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  #
#                                                                 #
                  CURRENTLY PROCESSING MODULE:
                         ★ regress ★
#                                                                 #
#  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  #
###################################################################

[Delegating control to module]

###################################################################
#  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  #
#                                                                 #
#  ⊗                CONFOUND REGRESSION MODULE                 ⊗  #
#                                                                 #
#  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  #
###################################################################

[I][/home/user/data/output/sub-005/run-01/sub-005_run-01.nii.gz]
[O][/home/user/data/output/sub-005/run-01/regress]
Design files  
2.000000

Current processing step:
Demeaning and detrending BOLD timeseries
····································································
· [Applying polynomial detrend]
· [Order: 2]
· [Executing detrend: Confound matrix]
····································································
Processing step complete:
Demeaning and detrending BOLD timeseries

Current processing step:
Temporally filtering image and confounds
····································································
· [butterworth filter]
· [High pass frequency: 0.06]
· [Low pass frequency: 0.125]
····································································
Processing step complete:
Temporally filtering image and confounds

Current processing step:
Converting BOLD timeseries to confound residuals
····································································
· [Computing confound correlations]
· [Executing detrend]
····································································
Processing step complete:
Converting BOLD timeseries to confound residuals

Current processing step:
Spatially filtering image
····································································
· [Configuring SUSAN]
· [Filter: susan]
· [Smoothing kernel: 6 mm]
INFO: /share/apps/singularity-3.5.2//matlab/startup.m does not exist ... creating
mkdir: cannot create directory ‘/share/apps/singularity-3.5.2//matlab’: Permission denied
touch: cannot touch '/share/apps/singularity-3.5.2//matlab/startup.m': No such file or directory
/opt/freesurfer/FreeSurferEnv.sh: line 231: /share/apps/singularity-3.5.2//matlab/startup.m: No such file or directory
/opt/freesurfer/FreeSurferEnv.sh: line 232: /share/apps/singularity-3.5.2//matlab/startup.m: No such file or directory
/opt/freesurfer/FreeSurferEnv.sh: line 233: /share/apps/singularity-3.5.2//matlab/startup.m: No such file or directory
/opt/freesurfer/FreeSurferEnv.sh: line 234: /share/apps/singularity-3.5.2//matlab/startup.m: No such file or directory
/opt/freesurfer/FreeSurferEnv.sh: line 235: /share/apps/singularity-3.5.2//matlab/startup.m: No such file or directory
/opt/freesurfer/FreeSurferEnv.sh: line 236: /share/apps/singularity-3.5.2//matlab/startup.m: No such file or directory
/opt/freesurfer/FreeSurferEnv.sh: line 237: /share/apps/singularity-3.5.2//matlab/startup.m: No such file or directory
/opt/freesurfer/FreeSurferEnv.sh: line 238: /share/apps/singularity-3.5.2//matlab/startup.m: No such file or directory
/opt/freesurfer/FreeSurferEnv.sh: line 239: /share/apps/singularity-3.5.2//matlab/startup.m: No such file or directory
/opt/freesurfer/FreeSurferEnv.sh: line 240: /share/apps/singularity-3.5.2//matlab/startup.m: No such file or directory
/opt/freesurfer/FreeSurferEnv.sh: line 241: /share/apps/singularity-3.5.2//matlab/startup.m: No such file or directory
/opt/freesurfer/FreeSurferEnv.sh: line 242: /share/apps/singularity-3.5.2//matlab/startup.m: No such file or directory
/opt/freesurfer/FreeSurferEnv.sh: line 243: /share/apps/singularity-3.5.2//matlab/startup.m: No such file or directory
/opt/freesurfer/FreeSurferEnv.sh: line 244: /share/apps/singularity-3.5.2//matlab/startup.m: No such file or directory
/opt/freesurfer/FreeSurferEnv.sh: line 245: /share/apps/singularity-3.5.2//matlab/startup.m: No such file or directory
/opt/freesurfer/FreeSurferEnv.sh: line 246: /share/apps/singularity-3.5.2//matlab/startup.m: No such file or directory
/opt/freesurfer/FreeSurferEnv.sh: line 247: /share/apps/singularity-3.5.2//matlab/startup.m: No such file or directory
grep: /share/apps/singularity-3.5.2//matlab/startup.m: No such file or directory
grep: /share/apps/singularity-3.5.2//matlab/startup.m: No such file or directory
grep: /share/apps/singularity-3.5.2//matlab/startup.m: No such file or directory
INFO: using NIfTI-1 qform 
mghRead(/home/user/data/sub-005/../../freesurfer/fsaverage5/mri/orig.mgz, -1): could not open file
srcvol = /home/user/data/output/sub-005/run-01/regress/boldtoMNI.nii.gz
srcreg unspecified
srcregold = 0
srcwarp unspecified
surf = white
hemi = lh
ProjFrac = 0.5
thickness = thickness
reshape = 0
interp = nearest
float2int = round
GetProjMax = 0
INFO: float2int code = 0
WARNING: niiRead(): unknown time units 0 in /home/user/data/output/sub-005/run-01/regress/boldtoMNI.nii.gz
Done loading volume
Computing registration from header.
  Using /home/user/data/sub-005/../../freesurfer/fsaverage5/mri/orig.mgz as target reference.
MRISread(/home/user/data/sub-005/../../freesurfer/fsaverage5/surf/lh.sphere): could not open file
No such file or directory
/opt/freesurfer/bin/mris_convert: could not read surface file /home/user/data/sub-005/../../freesurfer/fsaverage5/surf/lh.sphere
No such file or directory
INFO: using NIfTI-1 qform 
mghRead(/home/user/data/sub-005/../../freesurfer/fsaverage5/mri/orig.mgz, -1): could not open file
srcvol = /home/user/data/output/sub-005/run-01/regress/boldtoMNI.nii.gz
srcreg unspecified
srcregold = 0
srcwarp unspecified
surf = white
hemi = rh
ProjFrac = 0.5
thickness = thickness
reshape = 0
interp = nearest
float2int = round
GetProjMax = 0
INFO: float2int code = 0
WARNING: niiRead(): unknown time units 0 in /home/user/data/output/sub-005/run-01/regress/boldtoMNI.nii.gz
Done loading volume
Computing registration from header.
  Using /home/user/data/sub-005/../../freesurfer/fsaverage5/mri/orig.mgz as target reference.
MRISread(/home/user/data/sub-005/../../freesurfer/fsaverage5/surf/rh.sphere): could not open file
No such file or directory
/opt/freesurfer/bin/mris_convert: could not read surface file /home/user/data/sub-005/../../freesurfer/fsaverage5/surf/rh.sphere
No such file or directory

While running:
/xcpEngine/thirdparty/workbench/bin_rh_linux64/../exe_rh_linux64/wb_command -cifti-create-dense-scalar /home/user/data/output/sub-005/run-01/regress/sub-005_run-01_residual.dscalar.nii -left-metric /home/user/data/output/sub-005/run-01/regress/sub-005_run-01_residual_lh.func.gii -right-metric /home/user/data/output/sub-005/run-01/regress/sub-005_run-01_residual_rh.func.gii

ERROR: NAME OF FILE: sub-005_run-01_residual_lh.func.gii
PATH TO FILE: /home/user/data/output/sub-005/run-01/regress

File does not exist.

····································································
Processing step complete:
Spatially filtering image

Module complete

Runtime Information I was running singularity on slurm

Additional context I wonder if there is any issue with the design file or might be something wrong with the singularity images I am running.

[jasongong11]

• code to pull docker image:

  docker pull pennbbl/xcpengine

• code to build singularity image:

  sudo singularity build xcpengine_latest.simg docker://pennbbl/xcpengine

[jasongong11]

1. Looks like the error /share/apps/singularity-3.5.2//matlab/startup.m does not exist is an issue with the running on slurm HPC service. I rerun the code in a local machine using the same singularity image and got no error like this. Could you please offer some advise about how to get around with this? Or does this error matter?

2. Looks like the singularity binding point should be /data/derivatives rather than /data/derivatives/fmriprep, otherwise, the container could not find freesurfer dir.

3. Looks like there is a naming issue with /freesurfer/fsaverage5 which should be /freesurfer/fsaverage.

[jasongong11]

I have another question regarding how to compute pearson correlation adjacent matrix by own. Can I just extract time series from the file <sub>_<run>_img_sm6Std.nii.gz in norm dir using nilearn tool and compute the correlation matrix?

To be more specific, I want to run a dynamic connectivity network analysis. Which way is a better option: 1) to use the file in norm dir and separate different time windows and then extract time series and then compute correlation matrix by myself, or 2) simply separate the time series into different time window using the time series data ts.1D in fcon dir?

In other words, I think my question is that is the process to extract time series that is running in xcpengine is independent at each TR or is not independent. If each TR is independent, then I guess I could just separate the time series into different time windows using the ts.1D file. That would be much easier.

[a3sha2]

Hi @jasongong11

Yes the first issue is that you have new fmriprep output that doesn't include fsaverage5 in the FreeSurfer output. We have updated the xcpEngine, you can pull for the latest one, it will be fine.

you can include singularity run -cleanenv Clean the environment before running the container.

Remember <sub>_<run>_img_sm6Std.nii.gz has been spatially filtered after regression (_residualized.nii.gz). I would advise to extract timeseries in native space not norm. You can do it with nilearn tool or possibly with xcpengine ( check the last paragraph here: https://xcpengine.readthedocs.io/utils/roiquants.html#roi-quantification ). There are atlases in sub-*_atlas that have been registered to native space already after running xcpengine.

[jasongong11]

Hi @a3sha2 ,

Thank you very much for your response! The issues are gone with your advice! And I really appreciate your advice on extracting time series!

I am still puzzled about extracting time series by xcpengine. I noticed that after running the basic pipeline, the xcpengine already extracted time series based on Power et al. (2011) atlas. The result file is fcon/power264/<sub>_<run>_power264_ts.1D. I wonder if I can just use this file directly for further analysis?

Thanks again! Best, Jason

[a3sha2]

yes it can be use