Open HenrikEckermann opened 2 years ago
Thanks for creating the issue
Yes, thanks for the issue. Output from brm_multiple()
is indeed not supported by projpred and brms, at least when using the output from brm_multiple()
directly as input to get_refmodel()
. So this issue is about raising an appropriate message/warning/error. When running line refmodel2 <- get_refmodel(m2)
from your example, I already get
Warning message:
Using only the first imputed data set. Please interpret the results with caution until a more principled approach has been implemented.
For now, I would consider this as sufficient. I don't know if you got that warning, too. Might depend on the brms version you are using. I am using brms's current GitHub version (commit 8ba9e53). A Git Blame for the relevant lines in brms (https://github.com/paul-buerkner/brms/blob/8ba9e5333729e9d27e72a6fc81d746b4bb0d6251/R/brm_multiple.R#L231-L239 and https://github.com/paul-buerkner/brms/blob/8ba9e5333729e9d27e72a6fc81d746b4bb0d6251/R/prepare_predictions.R#L22) shows that brms should have thrown such a warning for about 3 years, but there might have been other lines I am currently not aware of which inhibited this or caused prepare_predictions.brmsfit()
not to be called in get_refmodel.brmsfit()
. Anyway, the important point is that the upcoming CRAN version of brms should throw this warning.
Thanks again and I'll think about documenting the missing support for brm_multiple()
.
Sorry, I should have left this open as a feature request! Re-opening now and adding the correct label.
Hi,
I tried to use projpred after multiple imputation using the mice package. It did not produce results as expected (see here for plots of the output. I suspected that this was due to a combination of lack of predictive power and because of multiple imputation outside of the statistical model with mice may not be supported. @avehtari confirmed to me that multiple imputation might not yet be supported and to open an issue here.
In case it is in fact supported already: Below code reproduces the output. I fit random noise with and without using multiple imputation. I would expect that RMSE is ~equal between models thanks to the horseshoe prior. This is the case without imputations but when using multiple imputations RMSE increases when more variables are included.
One more thing: Note that models have divergent transitions. In my real dataset I circumvented that by using different priors that are less regularizing (normal(0, 0.2) for all betas). I am aware that the horseshoe prior is important to prevent overfitting but I did not find a way yet to fit the models without divergent transitions when using it in my dataset (any hints are welcome if there are better ways to circumvent that).