search

compneuro-da / rsHRF

rsHRF: A Toolbox for Resting State HRF Deconvolution and Connectivity Analysis (MATLAB)
BSD 3-Clause "New" or "Revised" License
48 stars 15 forks source link

Deconvolving HRF step takes a huge amount of time #51

Closed lrq3000 closed 4 years ago

lrq3000 commented 6 years ago

The HRF estimation seems to be quite fast (particularly with microtime resolution 1 and AR off), but the "deconvolving HRF" step always takes very very long, even for just 300 volumes.

Is there a way to make this step faster?

danielemarinazzo commented 6 years ago

Thanks a lot for your message.

Could you please specify the architecture of the machine you are running the toolbox on? Also, can you please specify "very long"?

The Matlab SPM plugin takes a couple of minutes to run on a standard desktop pc for one subject in native space.

On Sun, Dec 2, 2018 at 2:00 AM Stephen L. notifications@github.com wrote:

The HRF estimation seems to be quite fast (particularly with microtime resolution 1 and AR off), but the "deconvolving HRF" step always takes very very long, even for just 300 volumes.

Is there a way to make this step faster?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/compneuro-da/rsHRF/issues/51, or mute the thread https://github.com/notifications/unsubscribe-auth/AFEKfr2pipgR_Nm_Zn4QV6ph3n_Jtjdtks5u0yYbgaJpZM4Y9LYd .

danielemarinazzo commented 6 years ago

Something else came to our minds, it might be that you do not use a mask, and then remove the outliers and replace these missing values (including a huge number of voxels outside the brain). this step can take very very long, and it's useless.

lrq3000 commented 6 years ago

Dear Daniele,

Thank you very much for your fast replies and also for your email! I take this opportunity to link to the solution to combine CONN and rshrf.

About the time it takes, I reworked further my pipeline to generate additional info that might be useful to you. Here is the log I get:

> -------------------------
> === HEMODYNAMIC RESPONSE FUNCTION USING RSHRF TOOLBOX ===
> 
> ---- PROCESSING CONDITION ket SUBJECT 1 (ana_5896) SESSION S2 ----
> 
> 
> ------------------------------------------------------------------------
> Running job #1
> ------------------------------------------------------------------------
> Running 'Voxel-wise HRF deconvolution'
> Reading NIFTI Data ...
> # 1739757 voxels inside defaut mask ...
> Reading Covariates ...
> Removing regressors & Detrend................. 
>  Frequency filter.................... 
>  #1739757 
> Starting parallel pool (parpool) using the 'local' profile ...
> connected to 6 workers.
> ...................................................................................................................................................................................................................................................
> Done HRF estimation   195.60 seconds
> Deconvolving HRF ...
> Elapsed time is 254.491797 seconds.
> Save HRF deconvolution results....
> height outlier: negative/zeros #243 + extreme #16801, [263.080 1315.501];
> Done    'Voxel-wise HRF deconvolution'
> Done
> 
> ---- PROCESSING CONDITION ket SUBJECT 1 (ana_5896) SESSION W1 ----
> 
> 
> ------------------------------------------------------------------------
> Running job #1
> ------------------------------------------------------------------------
> Running 'Voxel-wise HRF deconvolution'
> Reading NIFTI Data ...
> # 1658904 voxels inside defaut mask ...
> Reading Covariates ...
> Removing regressors & Detrend................. 
>  Frequency filter.................... 
>  #1658904 
> Starting parallel pool (parpool) using the 'local' profile ...
> connected to 6 workers.
> ...............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
> Done HRF estimation   185.79 seconds
> Deconvolving HRF ...
> Elapsed time is 88.535682 seconds.
> Save HRF deconvolution results....
> height outlier: negative/zeros #6353 + extreme #27171, [151.461 1348.671];
> Done    'Voxel-wise HRF deconvolution'
> Done
> 
> ---- PROCESSING CONDITION ket SUBJECT 2 (ana_5913) SESSION S2 ----
> 
> 
> ------------------------------------------------------------------------
> Running job #1
> ------------------------------------------------------------------------
> Running 'Voxel-wise HRF deconvolution'
> Reading NIFTI Data ...
> # 1473874 voxels inside defaut mask ...
> Reading Covariates ...
> Removing regressors & Detrend................. 
>  Frequency filter.................... 
>  #1473874 
> Starting parallel pool (parpool) using the 'local' profile ...
> connected to 6 workers.
> ............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
> Done HRF estimation   163.17 seconds
> Deconvolving HRF ...
> Elapsed time is 5416.796399 seconds.
> Save HRF deconvolution results....
> height outlier: negative/zeros #3459 + extreme #13968, [180.705 1833.208];
> Done    'Voxel-wise HRF deconvolution'
> Done
> 
> ---- PROCESSING CONDITION ket SUBJECT 2 (ana_5913) SESSION W1 ----
> 
> 
> ------------------------------------------------------------------------
> Running job #1
> ------------------------------------------------------------------------
> Running 'Voxel-wise HRF deconvolution'
> Reading NIFTI Data ...
> # 1431272 voxels inside defaut mask ...
> Reading Covariates ...
> Removing regressors & Detrend................. 
>  Frequency filter.................... 
>  #1431272 
> Starting parallel pool (parpool) using the 'local' profile ...
> connected to 6 workers.
> ...............................................................
> Done HRF estimation   142.70 seconds
> Deconvolving HRF ...
> Elapsed time is 22804.829655 seconds.
> Save HRF deconvolution results....
> height outlier: negative/zeros #63 + extreme #18985, [53.059 1353.533];
> Done    'Voxel-wise HRF deconvolution'
> Done
> 
> ---- PROCESSING CONDITION ket SUBJECT 3 (ana_5925) SESSION S2 ----
> 
> 
> ------------------------------------------------------------------------
> Running job #1
> ------------------------------------------------------------------------
> Running 'Voxel-wise HRF deconvolution'
> Reading NIFTI Data ...
> # 1578974 voxels inside defaut mask ...
> Reading Covariates ...
> Removing regressors & Detrend................. 
>  Frequency filter.................... 
>  #1578974 
> Starting parallel pool (parpool) using the 'local' profile ...
> connected to 6 workers.
> ............
> Done HRF estimation   101.43 seconds
> Deconvolving HRF ...

This is a post-processing on 9 subjects with 2 sessions (S2 and W1) for each. As you can see, the first deconvolution is quite quick, then for each session it becomes longer and longer, approximately 10x more time for each. I tried several times, modifying some parameters until I came back to near the default settings in rshrf, and it still happens the same, I never went beyond subject 3 session S2 (first session), because if the exponential increase is true, it would approximately take 50hours to complete (currently it's been running since 30 hours to no avail).

More precisely, here are the deconvolution runtimes so far from the log: 254s, 88s, 5416s, 22804s.

Your hypothesis that I don't use a mask is correct, but can this explain why the time is increasing between sessions? If it's because of the mask, I would expect the time increase to be constant for the same number of volumes and voxel resolution, which hold for my dataset.

I attach the jobs that my script is dynamically generating, maybe you can find more info from this? jobs.zip

I can also make available anything else if it can help, just let me know.

Thank you very much for taking the time to help me on this!

danielemarinazzo commented 6 years ago

Dear Stephan

thanks a lot for the update on the interface with CONN, I'm sure it will be useful to others.

The info that you provided indeed allows to understand that the issue is in the deconvolution phase.

We will look into the jobs.

For the moment, can you just try to run one subject/session per time, or change the order? Just to see if there is actually an increasing/accumulating delay (as it seems), or if some subjects/sessions are problematic.

lrq3000 commented 6 years ago

Thank you Daniele for this great idea, I'm running now the pipeline by beginning with the 3rd subject, so far it still did not finish, so it seems something in the data is making rshrf deconvolution go wild.

I let it run for the night to make sure and if it did not finish, tomorrow I can send you an excerpt of the dataset if you agree.

lrq3000 commented 6 years ago

BTW my pipeline is here: https://github.com/lrq3000/csg_mri_pipelines/blob/master/preprocessing/fmri/script_preproc_fmri_csg.m#L571

It's temporary as I will change it to process after CONN (currently it's done just before, which is wrong), but the rshrf parameters won't change.

guorongwu commented 6 years ago

matlabbatch{1}.spm.tools.HRF.vox_rsHRF.rmoutlier = 1; % remove outlier

matlabbatch{1}.spm.tools.HRF.vox_rsHRF.mask = {''}; % no mask included

matlabbatch{1}.spm.tools.HRF.vox_rsHRF.outdir = {''}; % save to current folder

I suggest you try to set .rmoutlier = 0;
.mask = {fullfile(spm(‘Dir’),’tpm’,'mask_ICV.nii')};

On 2 Dec 2018, at 09:00, Stephen L. <notifications@github.com mailto:notifications@github.com> wrote:

The HRF estimation seems to be quite fast (particularly with microtime resolution 1 and AR off), but the "deconvolving HRF" step always takes very very long, even for just 300 volumes.

Is there a way to make this step faster?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/compneuro-da/rsHRF/issues/51, or mute the thread https://github.com/notifications/unsubscribe-auth/ATmXf0YqoarbLxzA0kZrDnsquLQkJGXoks5u0yYbgaJpZM4Y9LYd.

lrq3000 commented 5 years ago

@guorongwu Thank you very much! Adding the mask indeed solved the issue (rmoutlier can be set to 1).

May I suggest to add this template as the default value in the rshrf batch?

I thought that rmoutlier was necessary, what would you guys advise for a typical resting state functional connectivity analysis? And what about AR correction?

Thank you very much! Best regards

guorongwu commented 5 years ago

in your pipeline, it's performed on normalized MNI space, I think you can add it as default value. I will also remove outlier, but if too much 'outlier' include, this step may 'change' the final deconvolved signals (similar to bad EEG channels for EEG analysis). we can use AR correction, I do not suggest you set a high AR order. the code may run faster if no AR correction. it will not achieve convergence in some signals, then without AR correction are better for these singals.

On Thu, Dec 6, 2018 at 5:14 AM Stephen L. notifications@github.com wrote:

@guorongwu https://github.com/guorongwu Thank you very much! Adding the mask indeed solved the issue (rmoutlier can be set to 1).

May I suggest to add this template as the default value in the rshrf batch?

I thought that rmoutlier was necessary, what would you guys advise for a typical resting state functional connectivity analysis? And what about AR correction?

Thank you very much! Best regards

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/compneuro-da/rsHRF/issues/51#issuecomment-444650879, or mute the thread https://github.com/notifications/unsubscribe-auth/ATmXf2eKio0X8yi9M8WbmNxQH25c7eWPks5u2Dc-gaJpZM4Y9LYd .

lrq3000 commented 5 years ago

Thank you very much @guorongwu for the info!

I am not sure I understand about the remove outlier option, if I enabled it (inpainting outliers with NaN) this can change the final deconvolved signal (or is it the other way around?).

For the default template, yes my pipeline is using MNI space, but since most SPM based pipelines do the same I think it would really ease newcomers to get a hang of the rshrf toolbox if this template was provided as the default, or if not, at least in the help message?

I have a final but very crucial question: can the rshrf toolbox work on smoothed data? Because the way CONN author advised to combine CONN + rshrf is to use rshrf after CONN denoising, which is done on smoothed data. If rshrf cannot work on smoothed data, would it be possible/mathematically meaningful to do the hrf deconvolution calculation on non-smoothed data, save it and apply it later on the smoothed data? What I have in mind is something like a normalization/DARTEL transform?

guorongwu commented 5 years ago

it only change the outlier signals (outlier detected from response height) actually, we include several demo jobs, which provide example to include different masks (native or MNI space). sure, you can use the smoothed data, it can increase SNR.

On Fri, Dec 7, 2018 at 9:45 AM Stephen L. notifications@github.com wrote:

Thank you very much @guorongwu https://github.com/guorongwu for the info!

I am not sure I understand about the remove outlier option, if I enabled it (inpainting outliers with NaN) this can change the final deconvolved signal (or is it the other way around?).

For the default template, yes my pipeline is using MNI space, but since most SPM based pipelines do the same I think it would really ease newcomers to get a hang of the rshrf toolbox if this template was provided as the default, or if not, at least in the help message?

I have a final but very crucial question: can the rshrf toolbox work on smoothed data? Because the way CONN author advised to combine CONN + rshrf is to use rshrf after CONN denoising, which is done on smoothed data. If rshrf cannot work on smoothed data, would it be possible/mathematically meaningful to do the hrf deconvolution calculation, save it and apply it on the smoothed data?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/compneuro-da/rsHRF/issues/51#issuecomment-445093852, or mute the thread https://github.com/notifications/unsubscribe-auth/ATmXf9EepuXYaxyL8FdKMqBa05wtvZHtks5u2cgvgaJpZM4Y9LYd .