search

ACCLAB / DABEST-python

Data Analysis with Bootstrapped ESTimation
https://acclab.github.io/DABEST-python/
Apache License 2.0
341 stars 47 forks source link

Are multi-group p-values corrected for multiple comparisons? #127

Closed charlesbmi closed 2 years ago

charlesbmi commented 2 years ago

Really enjoying using this package! I know this is a bit antithetical to estimation plots, but are the multi-group pvalue_students_t in .statistical_tests corrected for multiple comparisons when testing multiple groups?

Example:

>> diff.statistical_tests
  control test  control_N  test_N effect_size  is_paired  ...  pvalue_welch  statistic_welch  pvalue_students_t  statistic_students_t  pvalue_mann_whitney  statistic_mann_whitney
0      Xx   Rt         20      20   Hedges' g      False  ...      0.276593         1.105263           0.275995              1.105263             0.345422                   235.0
1      Xx   Ri         20      20   Hedges' g      False  ...      0.350531         0.946197           0.350027              0.946197             0.604458                   219.5
2      Xx   Rm         20      20   Hedges' g      False  ...      0.916870         0.105133           0.916823              0.105133             0.838272                   192.0
3      Xx   Rr         20      20   Hedges' g      False  ...      0.245580        -1.179686           0.245458             -1.179686             0.177002                   150.0
4      Xx   Rp         20      20   Hedges' g      False  ...      0.924163        -0.095829           0.924160             -0.095829             0.870344                   193.5
adamcc commented 2 years ago

Thanks Charles, glad you're enjoying!

As you guessed, because we don't believe in P values, we haven't done multiple-testing correction. If you're concerned about that you can do this with other software. I think if you use the non-parametric Kruskal-Wallis, it should be more conservative than the permutation P values (i.e. give higher P's in general).

Edit: I forgot to mention, if you are OK with "graphical NHST" you could also increase the CIs to increase stringency. For example, one of the MOE bars in a 99% CI will be closer to the null line, or more likely to touch it. Hope that makes sense.

charlesbmi commented 2 years ago

Sounds good, thanks for the help!