One- vs. two-tailed inference in simulation / numerical validation

[0todd0000]

(This issue is paraphrased from online workshop feedback)

The screenshot below shows three windows:

1. Two-sample t-test for 0D (simple scalar) data
2. Two-sample t-test for relatively smooth 1D data
3. Two-sample t-test for relatively rough 1D data

In each window:

• the left panel depicts random data (0D or 1D)
• the centre panel depicts the t-value (0D or 1D)
• the right panel depicts the distribution of the test statistic under the null hypothesis of no true effect

The paraphrased question is:

In the first window (with 0D data), both negative and positive t-values are shown in the distribution, but for the second and third windows (for 1D data), only positive t-values are shown. Thus it seems that the 0D analysis is two-tailed and the 1D analysis is one-tailed. Furthermore, the distribution curves are very similar between 0D and 1D, but the latter is cut-off at 0. However, it looked like that if the maximum absolute t-value was used for 1D, the distribution curves would have been exactly the same.

[0todd0000]

Apologies for the confusion! I can see why these results appear confusing, and that was due to a poorly conceived presentation, sorry about that! Here are some points to consider:

• All analyses are one-tailed
• It appears as if only positive values appear in the 1D cases, because the test statistic (the maximum t-value, or tmax) is almost always positive. tmax can indeed be less than zero, but only very rarely so. The distribution in Window 2 looks as if it is cut-off at zero only because smaller tmax values are exceedingly rare.
• The apparent similarity in the distributions stems from the mathematics of probability densities. Densities that are derived from the Gaussian distribution (including t, F, and chi-squared densities) look similar: they all exhibit an exponential decrease as the 90-to-95th percentile of the distribution is approached (i.e., the distribution's rightmost tail), and all densities converge to zero beyond that.
• If the maximum absolute t-value were used for the 0D case in Window 1, the distribution would indeed look a bit more similar to the 1D distribution in Window 2, but in that case, a fair comparison would require that Window 2 also consider the maximum absolute t-value; at the moment Window 2 considers only the maximum t-value (tmax) and not the minimum t-value. In other words, a fair comparison of the distributions requires that one chooses either one-tailed inference or two-tailed inference. This example chooses the latter, and one-tailed inference is presented in all Windows.