search

easystats / performance

:muscle: Models' quality and performance metrics (R2, ICC, LOO, AIC, BF, ...)
https://easystats.github.io/performance/
GNU General Public License v3.0
971 stars 87 forks source link

Questions about r2_loo() #190

Open jmgirard opened 3 years ago

jmgirard commented 3 years ago
  1. How concerned should I be about Pareto k warnings from r2_loo()?
  2. Is there a way to tweak the LOO settings within {performance} to address this or would I have to do the LOO separately (e.g., through brms::loo()) and then plug the results into some equation?
  3. Is there a way to apply LOO adjustment to the marginal R^2 in a mixed model?
strengejacke commented 3 years ago
  1. Not 100% sure, I would point you to help('pareto-k-diagnostic').
  2. We could add a ... argument that is passed down to loo::E_loo(). I must check, if e.g. threshold would resolve the warning.
  3. I would need to look deeper into the code. After quick look at r2_loo(), I would say it is possible to do this (not yet, but could be implemented). However, the approach I have in mind conflicts with #153. So I don't think this can be addressed before #153 is resolved.
jmgirard commented 3 years ago

Thanks as always.

  1. I'll dig into the literature to try to determine how concerned to be about Pareto k diagnostics.
  2. Note that I re-ran the {brms} model with save_pars = save_pars(all = TRUE) and then brms::loo(fit, moment_match = TRUE) resolved the issue, shrinking all the k values to below 0.7 (and therefore "ok" instead of "bad"). Not sure if loo::E_loo() has a similar argument, but worth looking into.
  3. Thank you! I really like the marginal/conditional R^2 distinction and would love to adjust the former with LOO.