stan-dev / projpred

Projection predictive variable selection
https://mc-stan.org/projpred/
Other
110 stars 26 forks source link

Warning for Pareto k-values in (0.5, 0.7] #451

Closed fweber144 closed 11 months ago

fweber144 commented 11 months ago

This amends #438 by incorporating Pareto k-values in the interval $(0.5, 0.7]$ (not just $> 0.7$) into the customized Pareto k-value warning message. Reasons:

  1. Consistency with the loo package (the loo package also differentiates $> 0.7$ further between $(0.7, 1]$ and $(1, \infty)$, but only in the detailed Pareto k-value table, not in the warning message).
  2. In general, the k-value threshold needs to be chosen depending on the Monte Carlo sample size (see the PSIS preprint). Within projpred, we often have small Monte Carlo sample sizes, especially in the PSIS-LOO CV that is performed in loo_varsel()'s validate_search = FALSE case. So it's probably safer to warn for Pareto k-values in the interval $(0.5, 0.7]$, too.