Two-sample t-test analysis

[RichLisonbee]

Hello Dr. Pataky,

I am performing a two-sample t-test and have run into some interesting results that I wanted to understand better. I have been performing the same type of analysis on other measurements with little to no issues. But I am now finding comparisons that would typically be statistically significant and are not.

For example.. I would expect that there would be more than just two small clusters when comparing these two sets of data.

Below is the code: spm = spm1d.stats.ttest2(data1,data2); spmi = spm.inference(0.05, 'two_tailed',true, 'interp',true);

Any help or guidance would be much appreciated. Please let me know if there is more information that you need.

[0todd0000]

There are two potential issues I see:

1. Smoothness: the data are smooth overall, but there appear to be high-frequency changes giving the means a step-like appearance in places. If you were to apply a low-pass filter to the data I'd expect the critical threshold to drop to 3.0 or possibly even lower.

2. Unequal variance: the Ad group appears to exhibit substantially more variability than the Control group. In cases of unequal variance like this, it can be deceiving to judge effect sizes in terms of the smaller variance. The overall variance is more important, so as a rough approximation imagine that the Ad variability magnitude also existed for the Control group. In this case the effect size would appear to be substantially smaller.

I think the combination of these two factors can explain the discrepancy between the expected and actual results.

As a third possibility:

3. Two-tailed vs. one-tailed analysis: these results are two tailed, but I wonder of a one-tailed analysis might be appropriate? If there is an a priori reason to expect that Ad would exhibit a higher mean than Control, then I think a one-tailed analysis could be appropriate. A one-tailed test by itself would lower the critical threshold to about 3.2.

[RichLisonbee]

Thank you so much!

These are issues we had considered but not fully investigated. It is good to get confirmation and we will investigate further. I will try applying a gaussian filter and see where that takes us. I am not well versed in stats and so this is a personal blind spot and I really appreciate your feedback and insight.

[ChantyR]

Hi Dr. Pataky,

I am also planning on running a very similar two-sample two-tailed t-test in MATLAB.

spm=spm1d.stats.ttest2(variableA, variableB); spmi=spm.inference(0.05,'two_tailed',true,'interp',true);

I was just wondering what the 'interp', true part of the code represents?

I found this definition under the one and two sample t-tests "interp : Interpolate clusters to the critical threshold (True {default} or False)." However, I also I found this under the MATLAB Documentation "The MATLAB version of spm1d does not interpolate supra-threshold clusters, thereby generating somewhat inaccurate measurements of cluster size."

Should I do spmi=spm.inference(0.05,'two_tailed',true,'interp',false); ?

Thank you very much for your help!

[0todd0000]

"interp" is a boolean (true or false) that specifies whether the endpoints of suprathreshold-clusters should be interpolated to the critical threshold (TRUE) or whether they should be integers (FALSE). Here "integers" refers to node numbers, usually from 1 to 100.

That MATLAB version warning regarding "interpolate" is actually no longer relevant; the MATLAB version also now interpolates suprathreshold cluster endpoints by default. After implementing the change in the MATLAB code I forgot to update the online documentation, so thank you for alerting me to the problem! I've now updated the documentation here: https://spm1d.org/DocumentationMatlab.html#label-documentationmatlab