Request for p.adjust correction override

[jimmyrigby94]

While I appreciate that emmeans provides intelligent corrections if users employ unthoughtful adjustment methods, it would be nice if there was an option that overrides the autocorrect. I am conducting some (not all) pairwise contrasts using contrast(), which should be an appropriate use of the Tukey correction if pairwise comparisons are examined post hoc (Maxwell, Dehlaney, & Kelly, 2017). The package defaults to sidak even when only pairwise comparisons are being examined! None of the below appraoches override the correction.

An autocorrect argument could be provide that defaults to TRUE, but can be changed to FALSE when the user insists on employing a different approach. To encourage proper uses of tests, a warning message could still be provided after the user overrides the default .

library(emmeans)
pigs.aov<-aov(conc ~ source, data = pigs)
emmeans.pigs<-emmeans(pigs.aov, "source")

mult_cont<-contrast(emmeans.pigs, 
                                  list(fishvsoy= c(1,-1,0),
                                  fishvskim= c(1,0,-1)),
                                  adjust = "tukey")

summary(mult_cont, adjust = "tukey")

[rvlenth]

You can already override the default correction method using the adjust argument with a valid method. Thus, providing an autocorrect argument would be completely redundant.

The reason adjust didn't work in your example is because the Tukey adjustment is not valid for the contrasts you specified. If you had specified adjust = "mvt" (which is what I suggest here) it would have worked. The mvt adjustment is the "exact" one-step adjustment based on the multivariate t distribution; this is the adjustment used by default in the multcomp package.

I do not have that Maxwell and Delaney reference, but I suggest that you read it again carefully. If they say something about using a Tukey adjustment for subsets in post hoc comparisons, I would bet money they are referring to a subset of the means, not to a subset of the comparisons. For example, If you have 4 means, it would be valid to do post hoc pairwise comparisons of just the means at levels 1, 3, and 4 (excluding level 2), using the Tukey adjustment for a set of 3 means. If instead you use the Tukey adjustment for pairwise comparisons based on 4 means, but apply that adjustment only to some subset of 3 of the 6 pairwise comparisons, that would be incredibly conservative. Using the Tukey adjustment based on 3 means for a set of comparisons that involves all 4 means would not necessarily protect the type I error probability.

[jimmyrigby94]

Hi Dr. Lenth,

Thank you for the response. I think that makes sense given the parameters for the q distribution (nmean and df). Essentially, when using a Tukey correction to test a subset of comparisons and setting nmean equal to k (the number of groups), we are correcting the p-values as if we were performing all pair-wise comparisons, ultimately reducing our power. Is that right?

Unfortunately, no - they referred to some comparisons instead of some means. Interesting! Here is the excerpt.

We recommend Tukey "when testing all pairwise and no complex comparisons (either planned or post hoc) or choosing to test only some pairwise comparisons post hoc." Admittedly, they only discuss a small subset of available corrections and never address mvt.

Do you recommend any simulation studies comparing the mvt with other adjustment methods?

Best, Jimmy

[rvlenth]

I'm disappointed that Maxwell et al. give that advice. It's not really totally wrong, in that it's guaranteed to be conservative; but it can be very conservative.

Maybe there's something about simulations or comparisons with other methods in the references provided in the multcomp package (one of its co-developers, Peter Westfall, is one of the world's leading experts on multiplicity); or for mvtnorm where the multivariate t algorithm is implemented. The mvt method really is exact when the underlying normality, etc. assumptions are met. Therefore, it's the same as the Tukey adjustment when using with all pairwise comparisons, and it's the same as the Dunnett adjustment when comparing all treatments with control (your example, BTW). The only glitch (and it's slight) is that a simulation method is used to do the computations; and for that reason, the adjusted P values may vary slightly if you repeat the analysis.