search

tramarobin / fctSnPM

Using spm1d package (v.0.4.3), compute anova and post-hoc tests from anova1 to anova3rm, with a non-parametric approach (permutation tests)
GNU General Public License v3.0
2 stars 2 forks source link

[JOSS Review] post hoc p values corrections? #9

Closed 0todd0000 closed 3 years ago

0todd0000 commented 3 years ago

Are post hoc probability values corrected (i.e., using the inverse Bonferroni correction)?

I can't seem to find this correction in the code.

For example, in the Speed.mat outputs for D1_ANOVA1rm.m I can find probability values in the clustersT cell array, but it is unclear whether these have been corrected.

Note that this is a minor issue, as the corrected p values will be alpha at most, but it's nevertheless important to ensure that p values aren't misleadingly low.

tramarobin commented 3 years ago

The outputs were reorganized and you can find the probabilities for each test and each cluster in the anova.mat file at anova.clusterP{test}(cluster) and in the posthoc.mat file at posthoc.tTests.clusterP{test}(cluster)

The probabilities are now corrected as suggested with the inverse Bonferroni correction for the ANOVA and the post hoc tests. It is indeed better if the values are not misleading.

0todd0000 commented 3 years ago

OK. I see, for example that Line 108 in fctPostHoc1d.m implements this:

posthoc.tTests.clusterP{comp}(c)=clustersT{c}.P*nComp;

This is fine, but now the problem with nT and alphaT becomes even more important. Now conceivably the user could specify nT as much smaller than nComp, and artificially lower p values. The relation amongst alpha, alphaT, nComp and nT is very confusing, and I am concerned that it may even be invalid. We have started to discuss this issue in #8, so I will paraphrase this comment there and close this issue.