rvlenth
commented
3 years ago OK, I think we've got it. Basic approach was to remember the by variables used whenever we construct paired comparisons (misc$estType = "pairs"), and make sure those haven't changed. In the rbind case, we changed misc$estType to "rbind" so it is never going to allow Tukey adjustments.
library(emmeans)
warp.lm = lm(breaks ~ wool * tension, data = warpbreaks)
EMM = emmeans(warp.lm, ~ tension | wool)
test(pairs(EMM), by = NULL)
## Note: adjust = "tukey" was changed to "sidak"
## because "tukey" is only appropriate for one set of pairwise comparisons
## contrast wool estimate SE df t.ratio p.value
## L - M A 20.556 5.16 48 3.986 0.0014
## L - H A 20.000 5.16 48 3.878 0.0019
## M - H A -0.556 5.16 48 -0.108 1.0000
## L - M B -0.556 5.16 48 -0.108 1.0000
## L - H B 9.444 5.16 48 1.831 0.3665
## M - H B 10.000 5.16 48 1.939 0.3030
##
## P value adjustment: sidak method for 6 tests
ALL = rbind(pairs(EMM, simple = "each"))
test(ALL, adjust = "tukey")
## Note: adjust = "tukey" was changed to "sidak"
## because "tukey" is only appropriate for one set of pairwise comparisons
## wool tension contrast estimate SE df t.ratio p.value
## A . L - M 20.556 5.16 48 3.986 0.0021
## A . L - H 20.000 5.16 48 3.878 0.0029
## A . M - H -0.556 5.16 48 -0.108 1.0000
## B . L - M -0.556 5.16 48 -0.108 1.0000
## B . L - H 9.444 5.16 48 1.831 0.4958
## B . M - H 10.000 5.16 48 1.939 0.4181
## . L A - B 16.333 5.16 48 3.167 0.0238
## . M A - B -4.778 5.16 48 -0.926 0.9817
## . H A - B 5.778 5.16 48 1.120 0.9398
##
## P value adjustment: sidak method for 9 testsCreated on 2021-05-03 by the reprex package (v1.0.0)
I also changed famSize returned by rbind to the number of estimates. Apparently I once allowed -- as in Case B above -- for a Tukey adjustment in rbind() results. Or maybe it is an artifact of the olden days when I also used famSize in the Scheffe adjustment, but now I compute the dimension of the estimates.
SchmidtPaul
Here are two situations where Tukey adjustments are allowed, but shouldn't be.
Case A
Here, we no longer have a family of pairwise comparisons (we have two such families combined into one). So the Tukey adjustment shouldn't be used. Worse, it defaults to Tukey. Even worse, it remembers the adjustment for one group, and uses that adjustment for all of them combined -- making this adjustment extremely liberal.
Case B
Continuing with the same objects...
Created on 2021-05-03 by the reprex package (v1.0.0) In this situation, at least the default isn't Tukey (it is Bonferroni, not shown). But it allows the Tukey adjustment, which it shouldn't. And note that it is a bit liberal: the P values are smaller than for
"mvt"which is the exact method.These errors happen because of remembered attributes leftover from the
pairs()calls, and failure to forget them when they don't apply. Case B is pretty easily fixed because we should just always haverbind()disable pairwise. Case A is trickier because we need to continue to allow Tukey when thebygroups are what they are supposed to be.