mlcollyer / RRPP

RRPP: An R package for fitting linear models to high-dimensional data using residual randomization
https://mlcollyer.github.io/RRPP/
4 stars 1 forks source link

p value correction for 'pairwise' and 'summary.pairwise' functions #3

Closed brikw closed 4 years ago

brikw commented 4 years ago

I am trying to use the pairwise function to test the absolute differences between slope vectors calculated for treatment groups on a covariate included in my model. This is for microbial community differences across plant samples. It seems like a p-value correction for multiple tests is being computed from the z-stats given for the pairwise comparisons, but I can't find mention of this anywhere in the help pages. Can someone comment on whether this is occurring or exactly how the p-values are being determined. See example output from a pairwise test below.

> summary(pwt.comm3, confidence = 0.95, test.type = "dist")
Pairwise comparisons
Groups:  C4grass monocot dicot soil 
RRPP: 1000 permutations
Slopes (vectors of variate change per one unit of covariate change, by group):
Vectors hidden (use show.vectors = TRUE to view)
Slope vector lengths
              C4grass    monocot      dicot       soil 
0.05132546 0.05132546 0.05132546 0.05132546 0.05132546 

Pairwise absolute difference (d) between vector lengths, plus statistics
                           d    UCL (95%)          Z Pr > d
:C4grass        0.000000e+00 6.938894e-18 -0.9474079 0.7640
:monocot        1.387779e-17 6.938894e-18  3.8880172 0.0025
:dicot          6.938894e-18 6.938894e-18  1.9418755 0.0570
:soil           1.387779e-17 1.040834e-17  3.0063474 0.0055
C4grass:monocot 1.387779e-17 6.938894e-18  3.8271141 0.0035
C4grass:dicot   6.938894e-18 6.938894e-18  1.9624340 0.0590
C4grass:soil    1.387779e-17 1.040834e-17  2.8623686 0.0100
monocot:dicot   6.938894e-18 6.938894e-18  1.7046905 0.0740
monocot:soil    0.000000e+00 6.938894e-18 -1.1387564 0.8370
dicot:soil      6.938894e-18 1.040834e-17  0.9712180 0.1955
mlcollyer commented 4 years ago

Brianna,

There is no P-value correction applied to P-values estimated from random permutations (an especially dangerous thing to do when the number of permutations could influence results).

It does, however, look like you had something illogical with your model (maybe no interactions between slopes and factors, as each has a vector of equal length. There can’t be an angular difference between parallel slopes, the kind you force when you have factors and a covariate, but no interaction between factors and the covariate.

Best, Mike

On Oct 1, 2020, at 4:19 PM, Briana K. Whitaker notifications@github.com wrote:

I am trying to use the pairwise function to test the absolute differences between slope vectors calculated for treatment groups on a covariate included in my model. This is for microbial community differences across plant samples. It seems like a p-value correction for multiple tests is being computed from the z-stats given for the pairwise comparisons, but I can't find mention of this anywhere in the help pages. Can someone comment on whether this is occurring or exactly how the p-values are being determined. See example output from a pairwise test below.

summary(pwt.comm3, confidence = 0.95, test.type = "dist") Pairwise comparisons Groups: C4grass monocot dicot soil RRPP: 1000 permutations Slopes (vectors of variate change per one unit of covariate change, by group): Vectors hidden (use show.vectors = TRUE to view) Slope vector lengths C4grass monocot dicot soil 0.05132546 0.05132546 0.05132546 0.05132546 0.05132546

Pairwise absolute difference (d) between vector lengths, plus statistics d UCL (95%) Z Pr > d :C4grass 0.000000e+00 6.938894e-18 -0.9474079 0.7640 :monocot 1.387779e-17 6.938894e-18 3.8880172 0.0025 :dicot 6.938894e-18 6.938894e-18 1.9418755 0.0570 :soil 1.387779e-17 1.040834e-17 3.0063474 0.0055 C4grass:monocot 1.387779e-17 6.938894e-18 3.8271141 0.0035 C4grass:dicot 6.938894e-18 6.938894e-18 1.9624340 0.0590 C4grass:soil 1.387779e-17 1.040834e-17 2.8623686 0.0100 monocot:dicot 6.938894e-18 6.938894e-18 1.7046905 0.0740 monocot:soil 0.000000e+00 6.938894e-18 -1.1387564 0.8370 dicot:soil 6.938894e-18 1.040834e-17 0.9712180 0.1955 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/mlcollyer/RRPP/issues/3, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABUNU4UQ7JAWE5GWNDE6QH3SITP4JANCNFSM4SA4PY2Q.