Temporal filtering failure due to NAs in the timeseries

[mfschmidt]

First, thanks for your time and work on xcpEngine. It was recently brought to my attention and is a really nice tool that I hope we can integrate into our normal workflows.

Describe the bug I ran xcpEngine on two subjects, each with two different smoothing levels. To do this, I downloaded the fc-36p_spkreg.dsn file and modified the regress_smo[3] value (one with 5, the other with 6). I also changed confound2_gsr[2] from 1 to 0. I believe the docker execution and design file are both good, because they worked for both runs of one subject, but failed with same error for both runs of another subject, described here.

This was run on fMRIPrep 21.1.1 output, which all looks OK to me and reports no errors. I viewed the /tmp/sub-P10023_ses-10023-regress-825485845~TEMP~_DMT.nii.gz file (due to the error message below) in fsleyes and don't see anything that would concern me, without knowing exactly what this file represents. It is a 4D file in MNI152NLin2009cAsym space, and every volume appears to contain real data.

I'm not sure where to investigate next. Any suggestions for where to look for the cause of the NAs, or more importantly, how to prevent them or clean them up?

Thank you for your help.

Cohort file Paste cohort file between the triple backticks

id0,id1,img
sub-P10023,ses-10023,func/sub-P10023_ses-10023_task-rest_run-01_space-MNI152NLin2009cAsym_res-2_desc-preproc_bold.nii.gz

Design File Paste your entire design (.dsn) file between the triple backticks

#!/usr/bin/env bash

###################################################################
#  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  #
###################################################################

###################################################################
# This design file stores the values of all variables required to
# execute a complete neuroimage processing pipeline. You may
# execute the analysis specified in this design file by calling
# (in any v4 or higher bash terminal):
#
# xcpEngine -d <design> -c <cohort> -o <output> <options>
#
# Variables fall into five general categories:
# * ANALYSIS VARIABLES are used at all stages of this analysis.
# * PIPELINE specifies the modules that comprise the analysis.
# * MODULE VARIABLES are used during one stage of the analysis.
#                  These are typically array variables with array
#                  indices equal to the index of the module that
#                  calls them.
# * OUTPUT VARIABLES may be used at all stages of the analysis.
#                  These are typically array variables with array
#                  indices equal to the value of the primary
#                  subject identifier. They will appear only in
#                  localised design files.
###################################################################

###################################################################
# ANALYSIS VARIABLES
###################################################################

analysis=fc_$(whoami)
design=/out/fc-36p_nogsr_spkreg_sm6.dsn
sequence=anatomical
standard=MNI%2x2x2

###################################################################
# PIPELINE
###################################################################

pipeline=prestats,confound2,regress,fcon,reho,alff,roiquant,norm,qcfc

###################################################################
# 1 PRESTATS
###################################################################

prestats_rerun[1]=1
prestats_cleanup[1]=1
prestats_process[1]=FMP

###################################################################
# 2 CONFOUND2
###################################################################

confound2_rps[2]=1
confound2_rms[2]=0
confound2_wm[2]=1
confound2_csf[2]=1
confound2_gsr[2]=0
confound2_acompcor[2]=0
confound2_tcompcor[2]=0
confound2_aroma[2]=0
confound2_past[2]=0
confound2_dx[2]=1
confound2_sq[2]=2
confound2_custom[2]=
confound2_censor[2]=1
confound2_censor_contig[2]=0
confound2_framewise[2]=fds:0.08,dv:2
confound2_rerun[2]=1
confound2_cleanup[2]=1

###################################################################
# 3  REGRESS
###################################################################

regress_tmpf[3]=butterworth
regress_hipass[3]=0.01
regress_lopass[3]=0.08
regress_tmpf_order[3]=1
regress_tmpf_pass[3]=2
regress_tmpf_ripple[3]=0.5
regress_tmpf_ripple2[3]=20
regress_dmdt[3]=2
regress_1ddt[3]=1
regress_smo[3]=6
regress_sptf[3]=susan
regress_usan[3]=default
regress_usan_space[3]=
regress_rerun[3]=0
regress_cleanup[3]=1
regress_process[3]=DMT-TMP-REG

###################################################################
# 4 FCON
###################################################################

fcon_atlas[4]=all
fcon_metric[4]=corrcoef
fcon_rerun[4]=0
fcon_cleanup[4]=1

###################################################################
# 5 REHO
###################################################################

reho_nhood[5]=vertices
reho_roikw[5]=0 # does nothing at the moment
reho_sptf[5]=susan
reho_smo[5]=6
reho_rerun[5]=0
reho_cleanup[5]=1

###################################################################
# 6 ALFF
###################################################################

alff_hipass[6]=0.01
alff_lopass[6]=0.08
alff_sptf[6]=susan
alff_smo[6]=6
alff_rerun[6]=0
alff_cleanup[6]=1

###################################################################
# 7 ROIQUANT
###################################################################

roiquant_atlas[7]=all
roiquant_globals[7]=1
roiquant_vol[7]=0
roiquant_rerun[7]=0
roiquant_cleanup[7]=1

###################################################################
# 8 NORM
###################################################################
norm_primary[8]=1
norm_rerun[8]=0
norm_cleanup[8]=1

##################################################################
# 9 QCFC
###################################################################
qcfc_atlas[9]=all
qcfc_sig[9]=fdr
qcfc_rerun[9]=0
qcfc_cleanup[9]=1

Error message Paste your error message between the backticks

Current processing step:
Temporally filtering image and confounds 
····································································
· [butterworth filter]
· [High pass frequency: 0.01]
· [Low pass frequency: 0.08]
· Interpolating over masked-out epochs...
· This will be slow 
/tmp/sub-P10023_ses-10023-regress-825485845~TEMP~_DMT.nii.gz
Voxel bin 1 out of 75
/xcpEngine/utils/interpolate.py:218: RuntimeWarning: invalid value encountered in true_divide
  norm_fac                =   std_recon/std_orig  
Voxel bin 2 out of 75
Voxel bin 3 out of 75
# ... same line for each voxel, 4 through 73
Voxel bin 74 out of 75
Voxel bin 75 out of 75
Error in na.omit.ts(x1, filt/a[1], sides = 1) :
  time series contains internal NAs
Calls: <Anonymous> ... filtfilt.default -> filter -> filter.default -> na.omit -> na.omit.ts
Execution halted 

· [A major error has occurred.]
· [The processing stream will now abort.]
· [Preparing diagnostics]
····································································

Runtime Information I ran xcpEngine via docker in a tmux session on Ubuntu Linux 22.04 as follows:

docker run --rm -it \
-v /var/data/fampath_upmc/derivatives/fmriprep_22.1.1/sub-P10023/ses-10023:/data:ro \
-v /var/data/fampath_upmc/derivatives/xcpengine_1.2.4/motion-36_wm_csf_nogsr_smooth-6:/out \
-v /tmp:/tmp \
-e USER='mike' \
pennbbl/xcpengine:latest \
-d /out/fc-36p_nogsr_spkreg_sm6.dsn \
-c /out/fampath_upmc_sample.csv \
-o /out \
-i /tmp \
-r /data

Additional context Add any other context about the problem here.