p_map - only for weak priors?

[mattansb]

Reading Mills's paper, it seems that the p-map is only suitable for weak / very diffused priors, as it is only under these conditions that the p-MAP has the "objective" qualities described in the paper.

I believe we've discussed this somewhere before (probably in the context of the effect of prior selection on the various indices), but as an example:

library(rstanarm)
library(bayestestR)

# flat prior
m1 <- stan_glm(extra ~ group, sleep,
               family = gaussian(),
               prior = NULL, refresh = 0)

# informed prior
m2 <- stan_glm(extra ~ group, sleep,
               family = gaussian(),
               prior = normal(0, 0.1), refresh = 0)

# informed prior 2
m3 <- stan_glm(extra ~ group, sleep,
               family = gaussian(),
               prior = normal(1, 0.1), refresh = 0)

p_map(m1)
#> # MAP-based p-value
#> 
#>    Parameter p_MAP
#>  (Intercept) 0.465
#>       group2 0.219
p_map(m2)
#> # MAP-based p-value
#> 
#>    Parameter   p_MAP
#>  (Intercept) 0.00605
#>       group2 0.96108
p_map(m3)
#> # MAP-based p-value
#> 
#>    Parameter  p_MAP
#>  (Intercept) 0.0761
#>       group2 0.0000

^{Created on 2019-11-13 by the reprex package (v0.3.0)}

(Again driving home the point that this index is affected by the prior, in the reverse Jeffreys-Lindley-Bartlett paradox.)

All this to say - I think we should add this as a strong suggestion in the functions documentation.

[DominiqueMakowski]

Agreed, and related to that we might wanna start thinking seriously about the follow-up paper, effect of priors specification on indices of existence & significance

[mattansb]

Yes to both (: - I would love to introduce to the world the reverse Jeffreys-Lindley-Bartlett paradox (aka the Ben-Shachar-Makowski-Lüdecke paradox; unfortunately, my name makes it sound like we are 4 people...)

[DominiqueMakowski]

in that case feel free to take the lead and create easystats/easystats/publications/bensachar2020 (hopefully 😅), in which we can start defining the simulation script 😉

[mattansb]

Seems like @strengejacke beat us too it with easystats\publications\ludecke_2020_priors?

[mattansb]

Hang on, I'm opening a real thread about this over on easystats.

[DominiqueMakowski]

if I'm not mistaken this is more an introduction and tutorial on priors specification for Bayesian models, based on one of his presentation, showing the effect that priors can have on the parameters, rather than per se a thorough investigation of the effect of priors on indices of sig and their sensitivity/robustness to them ☺️

[strengejacke]

unfortunately, my name makes it sound like we are 4 people...)

Maybe that's a good thing for naming paradoxes? ;-)

Seems like @strengejacke beat us too it with easystats\publications\ludecke_2020_priors?

Yes, like Dominique said, it is roughly a transcription of my presentation, and additionally a small simulation example, to support the tutorial character...