CI for mixed models R2 and ICC

[roaldarbol]

First of all, thanks for making all these packages - they're absolutely awesome! 👍

This is a feature idea. Would it be possible to provide confidence intervals for R2 and ICC for mixed models? As they are often used in biology, it would be super useful to get some easily accessible uncertainty estimates. The rptR package offers this, but not nearly to the breadth of models supported by performance. I can see that it is implemented for some R2's (though not for r2_nakagawa), but not really for ICCs.

[strengejacke]

Looking at the code from the rptR package, it looks like bootstrap is used to compute CIs, and I think this is the only way to compute CIs for ICC. However, it's time consuming...

[strengejacke]

For a possible implementation, I add this working example for later:

library(easystats)
#> # Attaching packages: easystats 0.5.2.13
#> ✔ bayestestR  0.13.0.1     ✔ correlation 0.8.3     
#> ✔ datawizard  0.6.3.3      ✔ effectsize  0.8.1.9999
#> ✔ insight     0.18.6.1     ✔ modelbased  0.8.5.1   
#> ✔ performance 0.10.0.6     ✔ parameters  0.19.0.14 
#> ✔ report      0.5.5.3      ✔ see         0.7.3.2
library(lme4)
#> Loading required package: Matrix
m <- lmer(Reaction ~ Days + (1 + Days | Subject), data = sleepstudy)
icc(m)$ICC_adjusted
#> [1] 0.7216603

icc_fun <- function(model, data, indices) {
  d <- data[indices, ] # allows boot to select sample
  fit <- suppressMessages(stats::update(model, data = d))
  icc(fit)$ICC_adjusted
}

res <- boot::boot(
  data = sleepstudy,
  statistic = icc_fun,
  R = 100,
  sim = "ordinary",
  model = m
)

out <- as.vector(res$t)
out <- out[!is.na(out)]
ci <- eti(out)

format_alert(sprintf("ICC: %.3g (%.3g, %.3g)", icc(m)$ICC_adjusted, ci$CI_low, ci$CI_high))
#> ICC: 0.722 (0.654, 0.878)

^{Created on 2022-10-27 with reprex v2.0.2}

[mattansb]

@strengejacke the bootstrapped CIs are not correct - you are sampling observations as if they were independent, but they are not (instead they are nested within the random grouping variables).

Instead of boot::boot() you should use lme4::bootMer().

[strengejacke]

Ok, should be fixed in #506. We might expand the bootstrapping feature by passing argument cluster, parallel and type to lme4::bootMer().

[strengejacke]

The boot::boot() is still available for those mixed models that don't work with lme4::bootMer().

[bwiernik]

With boot::boot() can we direct it to bootstrap at the cluster level?

[mattansb]

With boot::boot() can we direct it to bootstrap at the cluster level?

There's a strata argument, but I'm not sure what it does...

[bwiernik]

Does the opposite--resamples within groups (eg, to ensure that gender groups or experimental groups remain similarly represented)