Summary report: strange display of QCFC carpet plot

[julfou81]

Thank you for this beautiful tool! Describe the bug I'm using the "ready to use" (with no modification whatsoever) functional pipeline fc-36p.dsn (https://raw.githubusercontent.com/PennBBL/xcpEngine/master/designs/fc-36p.dsn) on data that has been pre-processed with fmriprep version 20.0.2. Cohort file

id0,img
sub-pilot1,fmriprep/sub-pilot1/func/sub-pilot1_task-AudioVisAssosFOV_run-01_space-T1w_desc-preproc_bold.nii.gz
sub-pilote1,fmriprep/sub-pilote1/func/sub-pilote1_task-VisuoTact_run-01_space-MNI152NLin2009cAsym_desc-preproc_bold.nii.gz
sub-pilote2,fmriprep/sub-pilote2/func/sub-pilote2_task-LocalizerMT_space-MNI152NLin2009cAsym_desc-preproc_bold.nii.gz

Design File

#!/usr/bin/env bash

###################################################################
#  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  ⊗  #
###################################################################

###################################################################
# This design file stores the values of all variables required to
# execute a complete neuroimage processing pipeline. You may
# execute the analysis specified in this design file by calling
# (in any v4 or higher bash terminal):
#
# xcpEngine -d <design> -c <cohort> -o <output> <options>
#
# Variables fall into five general categories:
# * ANALYSIS VARIABLES are used at all stages of this analysis.
# * PIPELINE specifies the modules that comprise the analysis.
# * MODULE VARIABLES are used during one stage of the analysis.
#                  These are typically array variables with array
#                  indices equal to the index of the module that
#                  calls them.
# * OUTPUT VARIABLES may be used at all stages of the analysis.
#                  These are typically array variables with array
#                  indices equal to the value of the primary
#                  subject identifier. They will appear only in
#                  localised design files.
###################################################################

###################################################################
# ANALYSIS VARIABLES
###################################################################

analysis=fc_$(whoami)
design=/home/jsein/data/fc-36p.dsn
sequence=anatomical
standard=MNI%2x2x2

###################################################################
# PIPELINE
###################################################################

pipeline=prestats,confound2,regress,fcon,reho,alff,roiquant,norm,qcfc

###################################################################
# 1 PRESTATS
###################################################################

prestats_rerun[1]=1
prestats_cleanup[1]=1
prestats_process[1]=FMP

###################################################################
# 2 CONFOUND2
###################################################################

confound2_rps[2]=1
confound2_rms[2]=0
confound2_wm[2]=1
confound2_csf[2]=1
confound2_gsr[2]=1
confound2_acompcor[2]=0
confound2_tcompcor[2]=0
confound2_aroma[2]=0
confound2_past[2]=0
confound2_dx[2]=1
confound2_sq[2]=2
confound2_custom[2]=
confound2_censor[2]=0
confound2_censor_contig[2]=0
confound2_framewise[2]=fds:0.167,dv:2
confound2_rerun[2]=0
confound2_cleanup[2]=1

###################################################################
# 3  REGRESS
###################################################################

regress_tmpf[3]=butterworth
regress_hipass[3]=0.01
regress_lopass[3]=0.08
regress_tmpf_order[3]=1
regress_tmpf_pass[3]=2
regress_tmpf_ripple[3]=0.5
regress_tmpf_ripple2[3]=20
regress_dmdt[3]=2
regress_1ddt[3]=1
regress_smo[3]=6
regress_sptf[3]=susan
regress_usan[3]=default
regress_usan_space[3]=
regress_rerun[3]=0
regress_cleanup[3]=1
regress_process[3]=DMT-TMP-REG

###################################################################
# 4 FCON
###################################################################

fcon_atlas[4]=all
fcon_metric[4]=corrcoef
fcon_thr[4]=N
fcon_pad[4]=FALSE
fcon_rerun[4]=0
fcon_cleanup[4]=1

###################################################################
# 5 REHO
###################################################################

reho_nhood[5]=vertices
reho_roikw[5]=0 # does nothing at the moment
reho_sptf[5]=susan
reho_smo[5]=6
reho_rerun[5]=0
reho_cleanup[5]=1

###################################################################
# 6 ALFF
###################################################################

alff_hipass[6]=0.01
alff_lopass[6]=0.08
alff_sptf[6]=susan
alff_smo[6]=6
alff_rerun[6]=0
alff_cleanup[6]=1

###################################################################
# 7 ROIQUANT
###################################################################

roiquant_atlas[7]=all
roiquant_globals[7]=1
roiquant_vol[7]=0
roiquant_rerun[7]=0
roiquant_cleanup[7]=1

###################################################################
# 8 NORM
###################################################################
norm_primary[8]=1
norm_rerun[8]=0
norm_cleanup[8]=1

##################################################################
# 9 QCFC
###################################################################
qcfc_atlas[9]=power264
qcfc_sig[9]=fdr
qcfc_rerun[9]=0
qcfc_cleanup[9]=1

Error message

no error to report

Runtime Information I was using the latest version of singularity

Additional context The pipeline worked well with all my subjects with no errors to report. However, the carpet plot for the residual BOLD data looks really strange for each subject even if the raw BOLD data plot looks good. Also, only the FD curve is visible on the raw data in bold, although DV, RMS, and FD curves are expected. Below is a screenshot of two carpet plots from a subject report. Do you know what can be done to solve this issue?

[a3sha2]

you have every set correct with design file as well. Pls can you post the the qc file prefix_quality.csv

[julfou81]

This is sub-pilote1_quality.csv id0,coregCrossCorr,coregJaccard,coregDice,coregCoverage,pctSpikesDV,nSpikesFD,pctSpikesFD,relMeanRMSMotion,nSpikesDV,relMaxRMSMotion,motionDVCorrInit,nNuisanceParameters,nVolCensored,normDice,normCoverage,normJaccard,normCrossCorr,estimatedLostTemporalDOF,motionDVCorrFinal sub-pilot1,0.8161626,0.7624626,0.8652242,1,0,181,.37396694214876033057,0.1681198,0,0.5247121,0.5823447,36,0,0.8501785,0.929132,0.7394003,0.8099919,217,0.2688544

[julfou81]

Also sub-pilot1_voxts.png looks strange:

[julfou81]

But prefix_voxts.png looks ok for some other subjects, even though the second carpet plot in he html report is still not looking as expected.

[a3sha2]

yes, the issue is that your bold has many spikes. You can see from quality csv number of volumes flagged is 181 (nSpikesFD). That is why you see estimatedLostTemporalDOF is very high (217=181+36, 36 is the number of nuisance regressors-nNuisanceParameters).

The FD threshold is very high, TR is of your bold is very small, you can see here https://xcpengine.readthedocs.io/modules/regress.html?highlight=framewise#temporal-censoring on how to change this confound2_framewise[2]=fds:0.167,dv:2 the best option is that you remove this line confound2_framewise[2]=fds:0.167,dv:2 from design file. sorry for late reply

[julfou81]

Thank you very much! It makes sense! Thank you for your explaination.I changed the line as you explained in the design file and the carpet plot was looking as expected. For an easier visual comparison with the capet plot of the unprocessed data, I also decided to remove the temporal filter by writing regress_process[cxt]=DMT-REG instead of regress_process[cxt]=DMT-TMP-REG. It gives me what I wanted.

However, along my many tests, there were several occasions when the summary report in HTML format was not generated, even though all the figures were present. There was no error reported in the process. I couldn't identify which condition triggers the generation of the report or not. Could you explain a bit more about it? Thank you in advance.

[a3sha2]

the figures may not generated completely. check the log files

[julfou81]

Indeed, there was an issue in the process and the corresponding figures were not generated and neither the report. It appeared when I wanted to modify the original 36p design and remove the GS as a confounds. On way to still get the report was to keep only the prestats, confounds, norm and QCFC modules in the design file:

pipeline=prestats,confound2,regress,norm,qcfc

The errors I found where:

Error in readNifti(roipath) : 
  Failed to read image from path /home/jsein/data/xcp_36p_noGS3/sub-pilote1/task-localizerPUFF/sub-pilote1_task-localizerPUFF_atlas/sub-pilote1_task-localizerPUFF_power264.nii.gz
Calls: readNifti -> .Call
In addition: Warning message:
In readNifti(roipath) :
  nifti_image_read: short header read '/home/jsein/data/xcp_36p_noGS3/sub-pilote1/task-localizerPUFF/sub-pilote1_task-localizerPUFF_atlas/sub-pilote1_task-localizerPUFF_power264.nii.gz'
Execution halted 

[...]

 Current processing step:
@3 Preparing summary graphics
····································································
· @3.1 Acquiring arguments
                             -   -   -
>> fslval /home/jsein/data/fmriprep/sub-pilote1/func/sub-pilote1_task-localizerPUFF_space-MNI152NLin2009cAsym_desc-preproc_bold.nii.gz pixdim4
                             -   -   -
>> 0.167*1.224000 
                             -   -   -
>> 0.083*1.224000 
/xcpEngine/modules/qcfc/qcfc.mod: line 177: ((: == 1 : syntax error: operand expected (error token is "== 1 ")
                             -   -   -
>> cp /home/jsein/data/xcp_36p_noGS3/sub-pilote1/task-localizerPUFF/qcfc/sub-pilote1_task-localizerPUFF~TEMP~_dvar.1D
cp: missing destination file operand after '/home/jsein/data/xcp_36p_noGS3/sub-pilote1/task-localizerPUFF/qcfc/sub-pilote1_task-localizerPUFF~TEMP~_dvar.1D'
Try 'cp --help' for more information.
                             -   -   -
>> cp /home/jsein/data/xcp_36p_noGS3/sub-pilote1/task-localizerPUFF/qcfc/sub-pilote1_task-localizerPUFF~TEMP~_fd.1D
cp: missing destination file operand after '/home/jsein/data/xcp_36p_noGS3/sub-pilote1/task-localizerPUFF/qcfc/sub-pilote1_task-localizerPUFF~TEMP~_fd.1D'
Try 'cp --help' for more information.
                             -   -   -
>> cp /home/jsein/data/xcp_36p_noGS3/sub-pilote1/task-localizerPUFF/qcfc/sub-pilote1_task-localizerPUFF~TEMP~_rms1.1D
cp: missing destination file operand after '/home/jsein/data/xcp_36p_noGS3/sub-pilote1/task-localizerPUFF/qcfc/sub-pilote1_task-localizerPUFF~TEMP~_rms1.1D'
Try 'cp --help' for more information.
· @3.2 Generating visuals
                             -   -   -
>> ${XCPEDIR}/utils/voxts.R -i /home/jsein/data/xcp_36p_noGS3/sub-pilote1/task-localizerPUFF/qcfc/sub-pilote1_task-localizerPUFF~TEMP~-pp-rs.nii.gz,/home/jsein/data/xcp_36p_noGS3/sub-pilote1/task-localizerPUFF/qcfc/sub-pilote1_task-localizerPUFF~TEMP~-dn-rs.nii.gz -r /home/jsein/data/xcp_36p_noGS3/sub-pilote1/task-localizerPUFF/qcfc/sub-pilote1_task-localizerPUFF~TEMP~-onion-rs.nii.gz -o /home/jsein/data/xcp_36p_noGS3/sub-pilote1/task-localizerPUFF/qcfc/sub-pilote1_task-localizerPUFF_voxts.png -n /xcpEngine/atlas/segmentation3/segmentation3NodeNames.txt
Error in file(file, "rt") : cannot open the connection
Calls: unlist -> read.table -> file
In addition: Warning message:
In file(file, "rt") : cannot open file 'NA': No such file or directory
Execution halted

Current processing step:
@4 Estimating loss of temporal degrees of freedom
····································································
                             -   -   -
>> rm -f /home/jsein/data/xcp_36p_noGS3/sub-pilote1/task-localizerPUFF/qcfc/sub-pilote1_task-localizerPUFF_tdof.txt
· @4.3 0 volumes censored
/xcpEngine/modules/qcfc/qcfc.mod: line 263: nSpikesDV: command not found 

[...]

 >> ${XCPEDIR}/utils/featureCorrelation.R -i /home/jsein/data/xcp_36p_noGS3/sub-pilote1/task-localizerPUFF/qcfc/sub-pilote1_task-localizerPUFF_dvars-std.1D,/home/jsein/data/xcp_36p_noGS3/sub-pilote1/task-localizerPUFF/qcfc/sub-pilote1_task-localizerPUFF~TEMP~_rms.1D
/usr/local/lib/python3.5/dist-packages/sklearn/externals/joblib/__init__.py:15: DeprecationWarning: sklearn.externals.joblib is deprecated in 0.21 and will be removed in 0.23. Please import this functionality directly from joblib, which can be installed with: pip install joblib. If this warning is raised when loading pickled models, you may need to re-serialize those models with scikit-learn 0.21+.
  warnings.warn(msg, category=DeprecationWarning)
Traceback (most recent call last):
  File "/xcpEngine/core/report.py", line 351, in <module>
    prefix+'_'+ii+'_ts.1D')
  File "/usr/local/lib/python3.5/dist-packages/numpy/lib/npyio.py", line 962, in loadtxt
    fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
  File "/usr/local/lib/python3.5/dist-packages/numpy/lib/_datasource.py", line 266, in open
    return ds.open(path, mode, encoding=encoding, newline=newline)
  File "/usr/local/lib/python3.5/dist-packages/numpy/lib/_datasource.py", line 624, in open
    raise IOError("%s not found." % path)
OSError: /home/jsein/data/xcp_36p_noGS3/sub-pilote1/task-localizerPUFF/fcon/schaefer100/sub-pilote1_task-localizerPUFF_schaefer100_ts.1D not found.
Sentry is attempting to send 0 pending error messages
Waiting up to 2 seconds
Press Ctrl-C to quit