Should the maximum statistic null distributions be separated by sign for two-sided tests?

[tsalo]

One question I have is whether the maximum statistic null distributions should be separated by sign for two-sided tests. For example, for the MKDAChi2, which can have positive and negative chi-squared values, should we retain maximum negative cluster size and maximum positive cluster size distributions separately?

I'm thinking about this primarily because of what I read in Albajes-Eizagirre et al. (2019):

SDM-PSI can currently save four different maximum statistics from the image of z-values: the largest z-value (i.e., voxel-based statistics), the maximum cluster size or mass after thresholding with a used-defined z- value (i.e., cluster-size or mass statistics) (Bullmore et al., 1999), and the maximum TFCE (Smith and Nichols, 2009). We have implemented TFCE in our software so that users who do not have FSL installed will still be able to use TFCE in SDM-PSI.

This procedure theoretically only tests hypothesis in one direction (e.g., patients > controls), but not the hypothesis in the other direction (e.g., patients < controls), and thus, the procedure should be conducted twice, one for each direction. However, given that the hypotheses are complementary and a permutation test is computationally consuming, SDM-PSI saves two numbers from each permutation: one for the positive hypothesis (e.g., the highest z-value) and one for the negative hypothesis (e.g., the lowest z-value), in two separate null distributions.

Originally posted by @tsalo in https://github.com/neurostuff/NiMARE/issues/655#issuecomment-1073029709

[tsalo]

I think this depends on whether the null distribution should be symmetric and zero-centered or not. For ALE subtraction analysis, any differences in the number of studies in the two datasets will shift the null distribution.

Is that going to be the case with MKDAChi2 as well?

Are there other cases where this should be considered?

[tsalo]

@nicholst is this issue something you're familiar with?