search

rvlenth / emmeans

Estimated marginal means
https://rvlenth.github.io/emmeans/
341 stars 30 forks source link

How does/should rbind interact with dunnettx adjustment? #447

Closed qdread closed 9 months ago

qdread commented 10 months ago

Hi Russ, I have a dataset where gene expression of many genes is being compared. There are several treatments and a control. We are only interested in comparing the mean of each treatment to the mean of the control, thus I am using the dunnettx adjustment. However, I think that the error rate should be controlled across all genes instead of separately within each gene (because the number of genes chosen to compare is more or less arbitrary, it would be easy to get false positives if lots and lots of genes are tested, even if we control for multiple comparisons vs. the control within each gene).

Where emmeans comes in is that I used rbind() to combine the contrasts for each gene together. This changes the dunnettx adjustment to a bonferroni adjustment by default. My question is: Is that the default behavior because it is the appropriate adjustment to use in this case? Or is there any better recommendation for what adjustment should be made if you are making many sets of treatment-vs-control comparisons, each of which should individually use the dunnettx adjustment?

Example code follows. If you need reproducible example please let me know.

fit_larvageneexp <- lmer(log(y) ~ cup_type + gene + cup_type:gene + (1 | grafting_round) + (1 | ID:grafting_round), data = larva_gene_long)
means_larvageneexp <- emmeans(fit_larvageneexp, trt.vs.ctrl ~ cup_type | gene, type = 'response')
rbind(means_larvageneexp$contrasts)

Before rbinding, each group of emmeans has P value adjustment: dunnettx method for 10 tests.
After rbinding, altogether it is P value adjustment: bonferroni method for 110 tests.

At risk of repeating myself, thanks again for this great package!

rvlenth commented 10 months ago

You have 11 sets of contrasts, and hence 11 controls -- not one -- are involved. The combined set of 110 tests is not the situation for which the Dunnett adjustment is appropriate, because it is not a family of comparisons with one control.

qdread commented 10 months ago

Thanks for responding. So you recommend that the Bonferroni adjustment be used in that case?

rvlenth commented 10 months ago

It's an appropriate default, but pretty conservative with that many tests. Don't people often use some FDR method in this application context? I can't really give good advice here.

rvlenth commented 9 months ago

Closing this issue as I think it's completed