search

FCP-INDI / C-PAC

Configurable Pipeline for the Analysis of Connectomes
https://fcp-indi.github.io/
GNU Lesser General Public License v3.0
62 stars 40 forks source link

update to fisher-z transform #310

Open maartenmennes opened 10 years ago

maartenmennes commented 10 years ago

I suggest to update the fisher-z transform

from:

corr_data = np.log((1 + corr_data) / (1 - corr_data)) / 2.0

to

corr_data = np.arctanh(corr_data) * np.sqrt(dof-3)

where dof = number of time-points used to calculate corr_data (len(timeseries_one_d) maybe?)

The arctanh is equal to the original transformation, but the important part is the addition of the normalisation by the variance of the distribution. This normalisation will update the scores to pseudo-z statistics, so you can just threshold them as you are used to (e.g., z > 2.3) or carry them forward to a group-level analysis. This way they also become comparable across studies with different scan lengths.

chrisgorgo commented 10 years ago

This is very interesting. Why -3? Shouldn't the DOF be also adjusted for the number of regressors in the nuisance regression and bandpass filtering? @juhuntenburg you might be interested in this.

ccraddock commented 10 years ago

The standard error for pearson's correlation is sqrt(n-3), so the goal here is to standardize to unit variance, see here: http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient

But the true DOF needs to be calculated based on the temporal autocorrelation of the data, and as Chris points out should be corrected for any DOF removed by nuisance regressions.

This is a huge problem with FC analyses, particularly when comparing data from different sites, and needs further inquiry.

We will keep it as a feature request, but I am not sure how we can handle this right now.

-cc

On Sun, Feb 16, 2014 at 4:49 PM, Chris Filo Gorgolewski < notifications@github.com> wrote:

This is very interesting. Why -3? Shouldn't the DOF be also adjusted for the number of regressors in the nuisance regression and bandpass filtering? @juhuntenburg https://github.com/juhuntenburg you might be interested in this.

Reply to this email directly or view it on GitHubhttps://github.com/FCP-INDI/C-PAC/issues/310#issuecomment-35215168 .

maartenmennes commented 10 years ago

Yes, I fully agree that it would be optimal to correct for the actual (true?) DOF! But in absence of that number, we are all happy to use n volumes as our DOF in any of the other single subject analyses we do. So why not for this one?

Using at least some sort of approximation to the DOF to do the normalization is better than not normalizing at all, because then the fisher-z transform is dependent on the actual number of timepoints that were used to calculate the correlation in the first place. This to me seems to be much less desirable than normalizing using some sort of approximation. No?

-mm

On Sun, Feb 16, 2014 at 11:46 PM, Cameron Craddock <notifications@github.com

wrote:

The standard error for pearson's correlation is sqrt(n-3), so the goal here is to standardize to unit variance, see here: http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient

But the true DOF needs to be calculated based on the temporal autocorrelation of the data, and as Chris points out should be corrected for any DOF removed by nuisance regressions.

This is a huge problem with FC analyses, particularly when comparing data from different sites, and needs further inquiry.

We will keep it as a feature request, but I am not sure how we can handle this right now.

-cc

On Sun, Feb 16, 2014 at 4:49 PM, Chris Filo Gorgolewski < notifications@github.com> wrote:

This is very interesting. Why -3? Shouldn't the DOF be also adjusted for the number of regressors in the nuisance regression and bandpass filtering? @juhuntenburg https://github.com/juhuntenburg you might be interested in this.

Reply to this email directly or view it on GitHub< https://github.com/FCP-INDI/C-PAC/issues/310#issuecomment-35215168>

.

Reply to this email directly or view it on GitHubhttps://github.com/FCP-INDI/C-PAC/issues/310#issuecomment-35218863 .

Maarten Mennes, Ph.D. Post-Doctoral Researcher Donders Institute for Brain, Cognition and Behaviour Dept. of Cognitive Neuroscience Radboud University Nijmegen Medical Centre Nijmegen The Netherlands

Google Scholar Author Linkhttp://scholar.google.com/citations?user=pLlSTVgAAAAJ&hl=en

ccraddock commented 10 years ago

Hey Maarten

I am not suggesting that we do not standardize the values. Just that sqrt(n-3) isn't a good way to standardize. If tr is the same across scans then N could be an alright to use, but when it differs there is problems. Instead I prefer z-score normalization where the variance is estimated from the data. I think that this is mostly adjusting for DOF and has been shown in (Yan et al 2012 neuroimage) to work well for multi site data. It is a shame that this paper didn't include the standardization that you suggested.

We will be implementing these different standardization in CPAC soon. We can add in your recommendation as well to enable future comparison.

Cameron

Sent from my iPhone

On Feb 17, 2014, at 6:37 AM, Maarten Mennes notifications@github.com wrote:

Yes, I fully agree that it would be optimal to correct for the actual (true?) DOF! But in absence of that number, we are all happy to use n volumes as our DOF in any of the other single subject analyses we do. So why not for this one?

Using at least some sort of approximation to the DOF to do the normalization is better than not normalizing at all, because then the fisher-z transform is dependent on the actual number of timepoints that were used to calculate the correlation in the first place. This to me seems to be much less desirable than normalizing using some sort of approximation. No?

-mm

On Sun, Feb 16, 2014 at 11:46 PM, Cameron Craddock <notifications@github.com

wrote:

The standard error for pearson's correlation is sqrt(n-3), so the goal here is to standardize to unit variance, see here: http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient

But the true DOF needs to be calculated based on the temporal autocorrelation of the data, and as Chris points out should be corrected for any DOF removed by nuisance regressions.

This is a huge problem with FC analyses, particularly when comparing data from different sites, and needs further inquiry.

We will keep it as a feature request, but I am not sure how we can handle this right now.

-cc

On Sun, Feb 16, 2014 at 4:49 PM, Chris Filo Gorgolewski < notifications@github.com> wrote:

This is very interesting. Why -3? Shouldn't the DOF be also adjusted for the number of regressors in the nuisance regression and bandpass filtering? @juhuntenburg https://github.com/juhuntenburg you might be interested in this.

Reply to this email directly or view it on GitHub< https://github.com/FCP-INDI/C-PAC/issues/310#issuecomment-35215168>

.

Reply to this email directly or view it on GitHubhttps://github.com/FCP-INDI/C-PAC/issues/310#issuecomment-35218863 .

Maarten Mennes, Ph.D. Post-Doctoral Researcher Donders Institute for Brain, Cognition and Behaviour Dept. of Cognitive Neuroscience Radboud University Nijmegen Medical Centre Nijmegen The Netherlands

Google Scholar Author Linkhttp://scholar.google.com/citations?user=pLlSTVgAAAAJ&hl=en — Reply to this email directly or view it on GitHub.

maartenmennes commented 10 years ago

Hello Cameron,

In your case, if N is the same across scans, there is no point in the standardisation, as all will be scaled equally. The standardisation would work across studies though (which is what I think you meant), but with the same problem you hint at. Therefore I think it does not matter whether all scans within a study have the same N, as N (and autocorrelation) will most likely differ between studies. Hell, autocorrelation will even be different between subjects within a study. So, no matter what N, we don't have a handle on the true DOF (~N) in the data at hand (Kundu might argue otherwise). Accordingly our scaling will be more/less off depending on the exact N and the autocorrelation present in the data.

That said, I still think that doing some form of standardisation is better than none. Given our universal acceptance of DOF = nTR I think sqrt(n-3) is a defendable choice. Importantly, it's a choice that will work for single subject data where you are not in the possibility of estimating the variance from the data (one could bootstrap or so, but still...)

Maarten

On Mon, Feb 17, 2014 at 1:55 PM, Cameron Craddock notifications@github.comwrote:

Hey Maarten

I am not suggesting that we do not standardize the values. Just that sqrt(n-3) isn't a good way to standardize. If tr is the same across scans then N could be an alright to use, but when it differs there is problems. Instead I prefer z-score normalization where the variance is estimated from the data. I think that this is mostly adjusting for DOF and has been shown in (Yan et al 2012 neuroimage) to work well for multi site data. It is a shame that this paper didn't include the standardization that you suggested.

We will be implementing these different standardization in CPAC soon. We can add in your recommendation as well to enable future comparison.

Cameron

Sent from my iPhone

On Feb 17, 2014, at 6:37 AM, Maarten Mennes notifications@github.com wrote:

Yes, I fully agree that it would be optimal to correct for the actual (true?) DOF! But in absence of that number, we are all happy to use n volumes as our DOF in any of the other single subject analyses we do. So why not for this one?

Using at least some sort of approximation to the DOF to do the normalization is better than not normalizing at all, because then the fisher-z transform is dependent on the actual number of timepoints that were used to calculate the correlation in the first place. This to me seems to be much less desirable than normalizing using some sort of approximation. No?

-mm

On Sun, Feb 16, 2014 at 11:46 PM, Cameron Craddock < notifications@github.com

wrote:

The standard error for pearson's correlation is sqrt(n-3), so the goal here is to standardize to unit variance, see here:

http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient

But the true DOF needs to be calculated based on the temporal autocorrelation of the data, and as Chris points out should be corrected for any DOF removed by nuisance regressions.

This is a huge problem with FC analyses, particularly when comparing data from different sites, and needs further inquiry.

We will keep it as a feature request, but I am not sure how we can handle this right now.

-cc

On Sun, Feb 16, 2014 at 4:49 PM, Chris Filo Gorgolewski < notifications@github.com> wrote:

This is very interesting. Why -3? Shouldn't the DOF be also adjusted for the number of regressors in the nuisance regression and bandpass filtering? @juhuntenburg https://github.com/juhuntenburg you might be interested in this.

Reply to this email directly or view it on GitHub< https://github.com/FCP-INDI/C-PAC/issues/310#issuecomment-35215168>

.

Reply to this email directly or view it on GitHub< https://github.com/FCP-INDI/C-PAC/issues/310#issuecomment-35218863> .

Maarten Mennes, Ph.D. Post-Doctoral Researcher Donders Institute for Brain, Cognition and Behaviour Dept. of Cognitive Neuroscience Radboud University Nijmegen Medical Centre Nijmegen The Netherlands

Google Scholar Author

Linkhttp://scholar.google.com/citations?user=pLlSTVgAAAAJ&hl=en

Reply to this email directly or view it on GitHub.

Reply to this email directly or view it on GitHubhttps://github.com/FCP-INDI/C-PAC/issues/310#issuecomment-35254305 .

Maarten Mennes, Ph.D. Post-Doctoral Researcher Donders Institute for Brain, Cognition and Behaviour Dept. of Cognitive Neuroscience Radboud University Nijmegen Medical Centre Nijmegen The Netherlands

Google Scholar Author Linkhttp://scholar.google.com/citations?user=pLlSTVgAAAAJ&hl=en

chrisgorgo commented 10 years ago

BTW don't you think it would be more elegant to to put the time series into the "nuisance" regression matrix coupled with a simple [1 0 0 0 ... 0] contrast (I think this is how PPI is doing this). This way we would be sure the DOF is estimated appropriately. Additionally one can implement bandpass correction as as set of regressors (SPM does that for task data). All signal and noise modelling would happen in one step!

maartenmennes commented 10 years ago

Putting the timeseries into a regression would allow differences in magnitude to play a role (they are 'ignored') in the correlation. However, I don't think this means the DOF is estimated appropriately automatically, some for of estimating the temporal autocorrelation would be needed (and is indeed implemented in the SPM and FSL glm implementations)

Bandpass filtering however, is so last year... ;)

Maarten

On Mon, Feb 17, 2014 at 3:10 PM, Chris Filo Gorgolewski < notifications@github.com> wrote:

BTW don't you think it would be more elegant to to put the time series into the "nuisance" regression matrix coupled with a simple [1 0 0 0 ... 0] contrast (I think this is how PPI is doing this). This way we would be sure the DOF is estimated appropriately. Additionally one can implement bandpass correction as as set of regressors (SPM does that for task data). All signal and noise modelling would happen in one step!

Reply to this email directly or view it on GitHubhttps://github.com/FCP-INDI/C-PAC/issues/310#issuecomment-35284436 .

Maarten Mennes, Ph.D. Post-Doctoral Researcher Donders Institute for Brain, Cognition and Behaviour Dept. of Cognitive Neuroscience Radboud University Nijmegen Medical Centre Nijmegen The Netherlands

Google Scholar Author Linkhttp://scholar.google.com/citations?user=pLlSTVgAAAAJ&hl=en

mnarayan commented 9 years ago

@maartenmennes I'm rather glad to find others in the field who recognize this problem. I am actively working on some solutions to resolve these issues in both correlation and partial correlation analyses over the next few months. Would be happy to work on this feature in the future, when we can make rigorously justify decisions.

Accounting for the sampling variability of each individual subject's functional connectivity is an extremely important issue. Whereas z-scoring across individual subjects address between subject variance, we need to ensure that subject level sample correlations are approximately N(0,1). While fisher's z-transform ensures that the distribution is approximately normal, it does not ensure that the asymptotic variance of the normal distribution is 1.

I've also noticed that resting state analyses in papers by Michael Fox typically standardize sample correlations by sqrt(t-3) with a Bartlett correction for autocorrelation per subject, in addition to applying Fisher's z-transform, where t is the number of within subject time-points. As the number of voxels or correlations increases, the lack of standardization via sqrt(t-3) is likely to inflate test statistics in downstream analyses. These kinds of errors due to multiple steps of statistical estimation is widely regarded as problematic in econometrics. While not accounting for autocorrelation makes this solution slightly wrong, I think standardizing by sqrt(t-3) is preferable to not doing so.