search

rmcelreath / rethinking

Statistical Rethinking course and book package
2.15k stars 603 forks source link

p value of the multi-level model (chapter 14) #255

Open SP977 opened 4 years ago

SP977 commented 4 years ago

Is it possible to get p-value of the model (multi-level models) built-in chapter 14? If yes, how we can do that?

bbolker commented 4 years ago

Which parameters/hypotheses precisely do you want p-values for?

SP977 commented 4 years ago

Is it possible for the whole model? If not---let say for any particular vector in the model?

SP977 commented 4 years ago

p-value here I am referring to model significant value, not the posterior value (just to avoid the confusion). I noticed some are reporting DIC and p-value of the model. Thanks

bbolker commented 4 years ago

Are you talking about posterior predictive p-values? https://www.tandfonline.com/doi/abs/10.1080/10705511.2018.1490648

For individual fixed-effect parameters, you could in principle use 2*min(p(beta<0),p(beta>0)) (i.e., double the smaller tail of the marginal posterior distribution); this is done by the MCMCglmm package (I don't know of a reference).

PS I don't have my copy of Rethinking handy, so apologies if my thoughts don't mesh perfectly with what's written there ...

SP977 commented 4 years ago

How we can estimate posterior predictive p-values (PPP) and deviance information criteria for the model? I am developing a model using ulam function in the rethinking package.

rmcelreath commented 4 years ago

Sorry I didn't notice this thread earlier. Been a crazy administrative season.

In general, it is not possible to get p-values for Bayesian models, because they do not necessarily make frequency claims.

I am only vaguely familiar with Bayesian quasi-p-values. I have never used them. But my understanding from Gelman et al is that they are tail probabilities of predictive distributions. Sorry I cannot be more helpful.

For DIC, I recommend you use WAIC instead, as described in the book. If you really want DIC, then you will have to do the calculations yourself unfortunately. If you can fit the model instead with the older map2stan function, it has an option to force the MAP values back through the model to compute DIC.

jamesbrownlow commented 4 years ago

Haben Sie überlegt Stochastic Differential Equation models für Zellevolution?

On Tue, Jun 30, 2020 at 5:22 AM Richard McElreath notifications@github.com wrote:

Sorry I didn't notice this thread earlier. Been a crazy administrative season.

In general, it is not possible to get p-values for Bayesian models, because they do not necessarily make frequency claims.

I am only vaguely familiar with Bayesian quasi-p-values. I have never used them. But my understanding from Gelman et al is that they are tail probabilities of predictive distributions. Sorry I cannot be more helpful.

For DIC, I recommend you use WAIC instead, as described in the book. If you really want DIC, then you will have to do the calculations yourself unfortunately. If you can fit the model instead with the older map2stan function, it has an option to force the MAP values back through the model to compute DIC.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/rmcelreath/rethinking/issues/255#issuecomment-651756697, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEBZLP2KKGSWT2SW7NVU3L3RZHKHVANCNFSM4OLLHFDQ .