Docker XCP performance optimization

[bdeck8317]

Describe the bug I am attempting to perform task regression on task data via XCP and re-running feat during processing. As it stands, XCP takes approximately 4 hours to run a single session for one subject. Most of that time is spent running feat. Is there any way to optimize the docker container to increase processing speed?

Cohort file

id0,id1,id2,id3,img,task_design
sub-DP5001,ses-NavonRealStim,NavonLow,run-1,fmriprep/sub-DP5001/ses-NavonRealStim/func/sub-DP5001_ses-NavonRealStim_task-NavonLow_run-1_space-T1w_desc-preproc_bold.nii.gz,data/task_designs/template.fsf

Design File Paste your entire design (.dsn) file between the triple backticks

#!/usr/bin/env bash

###################################################################
#  �^�^�  �^�^�  �^�^�  �^�^�  �^�^�  �^�^�  �^�^�  �^�^�  �^�^�  �^�^�  �^�^�  �^�^�  �^�^�  �^�^�  �^�^�  �^�^�  �^�^�  �^�^�  �^�^�  �^�^�  �^�^�  #
###################################################################

###################################################################
# This design file stores the values of all variables required to
# execute a complete neuroimage processing pipeline. You may
# execute the analysis specified in this design file by calling
# (in any v4 or higher bash terminal):
#
# xcpEngine <file>.dsn
#
# Variables fall into five general categories:
# * ANALYSIS VARIABLES are used at all stages of this analysis.
# * PIPELINE specifies the modules that comprise the analysis.
# * MODULE VARIABLES are used during one stage of the analysis.
#                  These are typically array variables with array
#                  indices equal to the index of the module that
#                  calls them.
# * OUTPUT VARIABLES may be used at all stages of the analysis.
#                  These are typically array variables with array
#                  indices equal to the value of the primary
#                  subject identifier. They will appear only in
#                  localised design files.
###################################################################

###################################################################
# ANALYSIS VARIABLES
###################################################################

analysis=task_root
design=/data/task_designs/task_fcon.dsn
sequence=anatomical
standard=MNI%2x2x2

###################################################################
# PIPELINE
###################################################################

pipeline=task,fcon,reho,alff,roiquant,norm

###################################################################
# 1 TASK
###################################################################

task_design[1]=template.fsf
task_fmriprep[1]=1
task_rpsd[1]=6
task_rerun[1]=Y
task_cleanup[1]=Y
task_vol2del[1]=0  # first N volume to be deleted -- none in this case
task_confound[1]='36p' # including acompcor parameters

###################################################################
# 2 FCON
###################################################################

fcon_atlas[2]=all
fcon_metric[2]=corrcoef
fcon_thr[2]=N
fcon_pad[2]=FALSE
fcon_rerun[2]=0
fcon_cleanup[2]=1

###################################################################
# 3 REHO
###################################################################

reho_nhood[3]=vertices
reho_roikw[3]=0 # does nothing at the moment
reho_sptf[3]=susan
reho_smo[3]=6
reho_rerun[3]=0
reho_cleanup[3]=1

###################################################################
# 4 ALFF
###################################################################

alff_hipass[4]=0.01
alff_lopass[4]=0.08
alff_sptf[4]=susan
alff_smo[4]=6
alff_rerun[4]=0
alff_cleanup[4]=1

###################################################################
# 5 ROIQUANT
###################################################################

roiquant_atlas[5]=all
roiquant_globals[5]=1
roiquant_vol[5]=0
roiquant_rerun[5]=0
roiquant_cleanup[5]=1

###################################################################
# 6 NORM
###################################################################

norm_rerun[6]=0
norm_cleanup[6]=1

Runtime Information

docker run -it \
    -v /Users/neurouser1/Documents/DP5/xcp/:/data \
    pennbbl/xcpengine:latest \
    -c /data/cohorts/DP5_task_cohort.csv \
    -d /data/task_designs/task_fcon.dsn \
    -o /data/xcpOutput/task_data/test \
    -i /data/xcpOutput/task_data/test/tmp/ \
    -r /data/fmriprep_out/

Additional context

[a3sha2]

that is too long , what the number of volume of the task? Also this task_confound[1]='36p' will increase the time during feat.

[bdeck8317]

@a3sha2 , thanks for getting back to me.

We have 960 volumes. We would like to keep the 36 parameter motion correction.

Any other suggestions to speed up processing?

[bdeck8317]

@a3sha2 @mattcieslak

Do either of you have suggestions for decreasing the processing speed here?

I.e. suggestions for what to set the cpu and memory for the container at?

[bdeck8317]

@a3sha2 , is there any way to speed up processing without removing the 36p?

How long does this normally take?

[a3sha2]

it depend on the data size. The average 5 min bold with 3mm resolution will require less than 10GB RAM and under 10 minutes. let me see the design file you are using!!!

[bdeck8317]

Hey @a3sha2 ,

Design file is the one above. We have 960 volumes with 36 p motion correction. Our bold scans are about 5 min with a pre/post design. So for each subject we have 2 images one for pre and one for post. Based on our specifications of our docker containers it was taking about 3 hours per image. This means it's 6 hours (3 hours each image). Feat then runs again based on our design file. So a total of 12 hours to run one subject.

This seems very long.

Would you have any recommendations for optimizing our cpu to ram ratio. Such as 8 cpu to 16 GB ram?

[bdeck8317]

@a3sha2

Would you have any recommendations for optimizing our cpu to ram ratio. Such as 8 cpu to 16 GB ram?

Thanks!