Cannot compute BF for 'brmsfit' models fit with default priors

[nahorp]

Hello :)

I am trying to calculate Bayes Factors in accordance with this informative blog post about how to do this with emmeans and bayestestR. The model I fit using brm is as below,

fit <- brm(y ~ A*B 
            + (A*B|subject),
            family = Gamma(link = "log"),
            data = df)

I then follow all the intermediate steps as in the blog post and when I come to the last step, using the describe_posterior function, I get the following error.

Computation of Bayes factors: sampling priors, please wait...
Error: Cannot compute BF for 'brmsfit' models fit with default priors.

I am unsure of why BF can't be computed for 'brmsfit' models with default priors?

Any insight in that regard would be appreciated. Thank you.

[mattansb]

Hi Rohan,

From the brms::set_prior documentation:

The default prior for population-level effects (including monotonic and category specific effects) is an improper flat prior over the reals.

It is not possible to sample from flat priors (try brm(Sepal.Length ~ Sepal.Width, iris, sample_prior = "only")).

But even if it was somehow possible, the resulting BFs would not be informative - in a flat distribution the probability/density of the null is by definition 0; and no matter the data, the null will have some non-0 probability/density (even if it is very small). So BF = 0/x = 0 - that is, very strong (the most extreme) evidence against the alternative... but what alternative? It wasn't specified...

Hope this helps!

[mattansb]

I've made the error a bit more informative:

Error: Cannot sample from flat priors (these are the default priors for fixed-effects in a 'brmsfit' model).

[DominiqueMakowski]

Maybe, instead of erroring, should we return 0 with the message as warning. Cause technically it's not really an error. We might add to the message a suggestion about what he should do to avoid this caveat:

"Warning: Flat priors (as set by default by brms) are not compatible with meaningful Bayes factors (favouring extreme evidence for the null). You should refit the model with informative priors."

[mattansb]

Cause technically it's not really an error.

It is an actual error - I've just wrapped around brms and rstanarm's errors.

Also, you know how I feel about defaults - what if the user misses the warning? (It's bad enough the methods for numeric and data.frame return BF=1 when no priors are provided!) I know we have different philosophies here: I think there is more value is an error letting the user know what they want is not possible, than there is in providing a faux result.

How's this for a more informative error:

Error: Cannot compute Bayes factors with flat priors (such as the default priors for fixed-effects in a 'brmsfit' model), as Bayes factor inform about the relative likelihood of two 'hypotheses', and flat priors provide no likelihood.

I don't think we should encourage setting priors in a post-hoc fashion (You should refit the model with informative priors.""). Maybe suggest some other methods of inference instead? (See Makowski et al, 2019 about inference with uninformative priors?)

[DominiqueMakowski]

It's your call! After all, even Yoda let Luke go to Bespin :)

You should refit the model with informative priors.

I was thinking of it as in "do your analysis with an informative prior in mind" rather than "mindlessly refit your model". IMO brms' default priors are rarely what the users actually want (but sometimes they might miss that specificity if they assume for instance rstanarm's behaviour)

[mattansb]

It's your call! After all, even Yoda let Luke go to Bespin :)

And look how much Luke learned by going there!

Any idea how to incorporate "do your analysis with an informative prior in mind" into the error?

[DominiqueMakowski]

".... You might want to do your analysis with informative priors."?

[mattansb]

Okay, I've added more info regarding flat priors to the functions' documentation, and I reference this info in the error itself.

I'm closing this issue.