[JOSS Review] Cluster p values

[0todd0000]

This discussion is continued from #8, which now focusses on alphaT. The discussion below pertains only to cluster p values as raised in #8.

Concerning the cluster probabilities. I modified it to taking into account the alpha risk and the number of comparisons. lines 107-110 of fctPostHoc1d, the cluster probabilities are corrected as clustersT{c}.P*(0.05/alpha). Is this correction ok ? Or should be taking into account only the number of comparisons and be clustersT{c}.P*nComp?

The inverse operation must be applied to calculated p values, so the latter appears to be correct.

• Line 56 in fctPostHoc1d applies alphaOriginal=alphaT/nComp; to obtain the critical p value, so the inverse operation (i.e., p_corrected = p_uncorrected*nComp) must be applied to correct p values.
  • Note also that alphaT should be removed as discussed in #8
  • Incidentially, I think p_critical may be a more appropriate variable name than alphaOriginal; to me, alphaOriginal suggests the alpha value chosen for the ANOVA.
  • If you agree, then Line 56 should probably be: p_critical = alpha / nComp

Consider the case of nComp = 2, for example. If alpha=0.05, then the critical threshold is p_critical = alpha / nComp which in this case implies p_critical = 0.025. If the test statistic continuum (i.e., t trajectory) in a post hoc test just touches this threshold, its (uncorrected) p value will be p_uncorrected = 0.025. However, just touching the threshold implies that it is on the threshold of rejecting the original ANOVA's null hypothesis, and the original ANOVA has an error rate of alpha=0.05. Thus you must apply the inverse of the critical-p operation (p_corrected = p_uncorrected * nComp;) to calculate corrected p values.

[tramarobin]

The clusters p-value have been corrected with inverse Bonferronni instead of the other method describe above. Line 56 was not modified but you can find these changes lines 97-98 posthoc.tTests.alphaOriginal=alphaT; (Note that the alphaT modification/suppression of issue #8 is in progress ) posthoc.tTests.pCritical=alpha;

The name of the variable for the alpha choosen by the user is still alphaOriginal and the alpha used after Bonferonni correction (or for the ANOVA to keep the same names) if now pCritical. This was modified on the online documentation and the function description as well.

[0todd0000]

OK.

Let's put this issue on hold, and resolve #8 first. I'd like to discuss alpha, alphaOriginal and alphaAOV, but #8 is currently a bigger issue that may affect this discussion. After your modifications and/or rebuttals regarding #8, let's re-discuss this issue.

[0todd0000]

This issue is resolved by our discussions in #8, so I will close it.

In case you'd like to tidy up all alpha variables: alphaAOV appears in the function signatures for various functions including fctPostHoc1d.m, for example, but alphaAOV is not documented. Since users are not meant to use these functions (or alphaAOV) directly, this is not a problem. Please feel free to keep alphaAOV or use alpha, whichever you prefer for code development and maintenance purposes.