Segmentation faults at various stages of processing

[dlevitas]

Describe the bug I'm new to xcpEngine and have been trying to set things up with some data of mine, following fMRIPrep pre-processing.

Cohort file

sub,task,run,img
sub-1000,std,1,sub-1000/func/sub-1000_task-std_run-1_space-MNI152NLin6Asym_desc-smoothAROMAnonaggr_bold.nii.gz
sub-1000,std,2,sub-1000/func/sub-1000_task-std_run-2_space-MNI152NLin6Asym_desc-smoothAROMAnonaggr_bold.nii.gz
sub-1000,std,3,sub-1000/func/sub-1000_task-std_run-3_space-MNI152NLin6Asym_desc-smoothAROMAnonaggr_bold.nii.gz
sub-1000,std,4,sub-1000/func/sub-1000_task-std_run-4_space-MNI152NLin6Asym_desc-smoothAROMAnonaggr_bold.nii.gz
sub-1000,std,5,sub-1000/func/sub-1000_task-std_run-5_space-MNI152NLin6Asym_desc-smoothAROMAnonaggr_bold.nii.gz
sub-1000,std,6,sub-1000/func/sub-1000_task-std_run-6_space-MNI152NLin6Asym_desc-smoothAROMAnonaggr_bold.nii.gz
sub-1000,std,7,sub-1000/func/sub-1000_task-std_run-7_space-MNI152NLin6Asym_desc-smoothAROMAnonaggr_bold.nii.gz
sub-1000,std,8,sub-1000/func/sub-1000_task-std_run-8_space-MNI152NLin6Asym_desc-smoothAROMAnonaggr_bold.nii.gz

Design File Paste your entire design (.dsn) file between the triple backticks

#!/usr/bin/env bash

###################################################################
#  ??  ??  ??  ??  ??  ??  ??  ??  ??  ??  ??  ??  ??  ??  ??  ??  ??  ??  ??  ??  ??  #
###################################################################

###################################################################
# This design file stores the values of all variables required to
# execute a complete neuroimage processing pipeline. You may
# execute the analysis specified in this design file by calling
# (in any v4 or higher bash terminal):
#
# xcpEngine -d <design> -c <cohort> -o <output> <options>
#
# Variables fall into five general categories:
# * ANALYSIS VARIABLES are used at all stages of this analysis.
# * PIPELINE specifies the modules that comprise the analysis.
# * MODULE VARIABLES are used during one stage of the analysis.
#                  These are typically array variables with array
#                  indices equal to the index of the module that
#                  calls them.
# * OUTPUT VARIABLES may be used at all stages of the analysis.
#                  These are typically array variables with array
#                  indices equal to the value of the primary
#                  subject identifier. They will appear only in
#                  localised design files.
###################################################################

###################################################################
# ANALYSIS VARIABLES
###################################################################

analysis=fc_$(whoami)
design=/N/slate/dlevitas/xcpEngine-master/designs/fc-24p_gsr.dsn
sequence=anatomical
standard=MNI%2x2x2

###################################################################
# PIPELINE
###################################################################

pipeline=prestats,confound2,regress,fcon,reho,alff,roiquant,norm,qcfc

###################################################################
# 1 PRESTATS
###################################################################

prestats_rerun[1]=1
prestats_cleanup[1]=1
prestats_process[1]=FMP

###################################################################
# 2 CONFOUND2
###################################################################

confound2_rps[2]=1
confound2_rms[2]=0
confound2_wm[2]=0
confound2_csf[2]=0
confound2_gsr[2]=1  
confound2_acompcor[2]=0
confound2_tcompcor[2]=0
confound2_aroma[2]=0
confound2_past[2]=0
confound2_dx[2]=1
confound2_sq[2]=2
confound2_custom[2]=
confound2_censor[2]=0
confound2_censor_contig[2]=0
confound2_rerun[2]=0
confound2_cleanup[2]=1

###################################################################
# 3  REGRESS
###################################################################

regress_tmpf[3]=butterworth
regress_hipass[3]=0.01
regress_lopass[3]=0.08
regress_tmpf_order[3]=1
regress_tmpf_pass[3]=2
regress_tmpf_ripple[3]=0.5
regress_tmpf_ripple2[3]=20
regress_dmdt[3]=2
regress_1ddt[3]=1
regress_smo[3]=6
regress_sptf[3]=susan
regress_usan[3]=default
regress_usan_space[3]=
regress_rerun[3]=0
regress_cleanup[3]=1
regress_process[3]=DMT-TMP-REG

###################################################################
# 4 FCON
###################################################################

fcon_atlas[4]=all
fcon_metric[4]=corrcoef
fcon_rerun[4]=0
fcon_cleanup[4]=1

###################################################################
# 5 REHO
###################################################################

reho_nhood[5]=vertices
reho_roikw[5]=0 # does nothing at the moment
reho_sptf[5]=susan
reho_smo[5]=6
reho_rerun[5]=0
reho_cleanup[5]=1

###################################################################
# 6 ALFF
###################################################################

alff_hipass[6]=0.01
alff_lopass[6]=0.08
alff_sptf[6]=susan
alff_smo[6]=6
alff_rerun[6]=0
alff_cleanup[6]=1

###################################################################
# 7 ROIQUANT
###################################################################

roiquant_atlas[7]=all
roiquant_globals[7]=1
roiquant_vol[7]=0
roiquant_rerun[7]=0
roiquant_cleanup[7]=1

###################################################################
# 8 NORM
###################################################################
norm_primary[8]=1
norm_rerun[8]=0
norm_cleanup[8]=1

##################################################################
# 9 QCFC
###################################################################
qcfc_atlas[9]=power264
qcfc_sig[9]=fdr
qcfc_rerun[9]=0
qcfc_cleanup[9]=1

Error message Paste your error message between the backticks

Current processing step:
getting data from fmriprep directory
·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?
·? checking refvolume and structural orientation
 file /N/slate/dlevitas/dan_xcpEngine_test_dataset/derivatives/xcpEngine-1.2.3/sub-1000/prestats/sub-1000_referenceVolume.nii.gz does not exist . 
/xcpEngine/core/functions/exec_ants: line 1:  6536 Segmentation fault      ${ANTSPATH}/"${@}"
 file /N/slate/dlevitas/dan_xcpEngine_test_dataset/derivatives/xcpEngine-1.2.3/sub-1000/prestats/sub-1000_struct.nii.gz does not exist . 
 file /N/slate/dlevitas/dan_xcpEngine_test_dataset/derivatives/xcpEngine-1.2.3/sub-1000/prestats/sub-1000_referenceVolume.nii.gz does not exist . 
/xcpEngine/core/functions/exec_ants: line 1:  6564 Segmentation fault      ${ANTSPATH}/"${@}"
 file /N/slate/dlevitas/dan_xcpEngine_test_dataset/derivatives/xcpEngine-1.2.3/sub-1000/prestats/sub-1000_referenceVolume.nii.gz does not exist . 
/xcpEngine/core/functions/exec_ants: line 1:  6594 Segmentation fault      ${ANTSPATH}/"${@}"
·? generate mask and referenceVolumeBrain
 file /N/slate/dlevitas/dan_xcpEngine_test_dataset/derivatives/xcpEngine-1.2.3/sub-1000/prestats/sub-1000_referenceVolume.nii.gz does not exist . 
/xcpEngine/core/functions/exec_ants: line 1:  6623 Segmentation fault      ${ANTSPATH}/"${@}"
Cannot open volume /N/slate/dlevitas/dan_xcpEngine_test_dataset/derivatives/fmriprep-20.2.0/sub-1000/func/sub-1000_task-std_run-1_space-MNI152NLin6Asym_desc-brain_mask for reading!
 file /N/slate/dlevitas/dan_xcpEngine_test_dataset/derivatives/xcpEngine-1.2.3/sub-1000/prestats/sub-1000_mask.nii.gz does not exist . 
 file /N/slate/dlevitas/dan_xcpEngine_test_dataset/derivatives/xcpEngine-1.2.3/sub-1000/prestats/sub-1000_referenceVolume.nii.gz does not exist . 
/xcpEngine/core/functions/exec_ants: line 1:  6653 Segmentation fault      ${ANTSPATH}/"${@}"
Cannot open volume /N/slate/dlevitas/dan_xcpEngine_test_dataset/derivatives/xcpEngine-1.2.3/sub-1000/prestats/sub-1000_mask for reading!
Cannot open volume /N/slate/dlevitas/dan_xcpEngine_test_dataset/derivatives/xcpEngine-1.2.3/sub-1000/prestats/sub-1000_mask for reading!
source space map '/N/slate/dlevitas/dan_xcpEngine_test_dataset/derivatives/xcpEngine-1.2.3/sub-1000/prestats/sub-1000_struct.nii.gz' is not an image
·? Quality assessment
Error in readNifti(mask1path) : 
  Failed to read image from path /N/slate/dlevitas/dan_xcpEngine_test_dataset/derivatives/xcpEngine-1.2.3/sub-1000/prestats/sub-1000_segmentation.nii.gz
Calls: readNifti -> .Call
In addition: Warning message:
In readNifti(mask1path) :
  nifti_image_read: failed to find header file for '/N/slate/dlevitas/dan_xcpEngine_test_dataset/derivatives/xcpEngine-1.2.3/sub-1000/prestats/sub-1000_segmentation.nii.gz'
Execution halted

...

Current processing step:
Demeaning and detrending BOLD timeseries
·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?·?
·? [Applying polynomial detrend]
·? [Order: 2]

 *** caught segfault ***
address 0x20, cause 'memory not mapped'

Traceback:
 1: .Call("asNifti", x, reference, datatype, internal, PACKAGE = "RNifti")
 2: asNifti.default(image, internal = TRUE)
 3: asNifti(image, internal = TRUE)
 4: dumpNifti(impath)
An irrecoverable exception occurred. R is aborting now ...
/xcpEngine/core/functions/exec_xcp: line 1:  2637 Segmentation fault      ${XCPEDIR}/utils/"${@}"

·? [A major error has occurred.]
·? [The processing stream will now abort.]
·? [Preparing diagnostics]

Runtime Information Running via Singularity (version 3.6.4). xcpEngine version is 1.2.3. Command I am using is:

unset PYTHONPATH; singularity run -B /N/slate/$(whoami):/N/slate/$(whoami) \
    /N/dcwan/projects/irf/containers/xcpEngine-${xcpEngine_version}.simg \
    -d ${design_file} \
    -c ${bids_root_dir}/derivatives/xcpEngine-${xcpEngine_version}/${processing}_cohort.csv \
    -i $bids_root_dir/derivatives/xcpEngine-${xcpEngine_version}/temp \
    -r $fmriprep_data_dir \
    -o $output_dir

Additional context I've been running xcpEngine through my institution's HPC, submitting the job with up to 30gb of vmem to ensure my issue isn't related to memory. Changing the file permissions hasn't resolved the problem either. Hoping for some guidance on this.

Thanks!

[dlevitas]

These issues seem to have been resolved due to some trouble shooting, though I can't recall specifically what I did to fix this. My apologizes for not documenting my work better.