stan-dev / projpred

Projection predictive variable selection
https://mc-stan.org/projpred/
Other
110 stars 26 forks source link

Fix seeds/PRNG usage #412

Closed fweber144 closed 1 year ago

fweber144 commented 1 year ago

This is a major fix for PRNG usage within projpred, see the NEWS.md entries added here. The commit messages provide more details.

With this PR, we also make use of the fix in the vignettes: Now, we only need to set a seed once at the beginning (except for the comparison of latent and traditional projection, but this has nothing to do with the seeds issue fixed here).

fweber144 commented 1 year ago

I forgot to add a short illustration of the issue and the fix for it:

data("df_gaussian", package = "projpred")
df_gaussian <- df_gaussian[1:29, ]
dat <- data.frame(y = df_gaussian$y, df_gaussian$x)
library(rstanarm)
rfit <- stan_glm(y ~ X1 + X2 + X3 + X4 + X5,
                 data = dat,
                 chains = 1,
                 iter = 500,
                 seed = 1140350788,
                 refresh = 0)

### Use the installed version 2.5.0 (or load it via devtools::load_all()):
library(projpred)
###
### Load a version > 2.5.0 containing the fix for this seed issue (or
### attach it via `library(projpred)` if installed):
# devtools::load_all()
###

set.seed(467845213)
vs1 <- varsel(rfit,
              nclusters = 23,
              nclusters_pred = 25)
vs2 <- varsel(rfit,
              nclusters = 23,
              nclusters_pred = 25)
smmry1 <- summary(vs1)
smmry1 <- smmry1[setdiff(names(smmry1), "family")]
smmry2 <- summary(vs2)
smmry2 <- smmry2[setdiff(names(smmry2), "family")]
identical(smmry1, smmry2)
## --> `TRUE` with projpred 2.5.0, `FALSE` with a projpred version > 2.5.0
## containing the fix for this seed issue.
vs1 <- vs1[setdiff(names(vs1), c("refmodel", "search_path"))]
vs2 <- vs2[setdiff(names(vs2), c("refmodel", "search_path"))]
identical(vs1, vs2)
## --> `TRUE` with projpred 2.5.0, `FALSE` with a projpred version > 2.5.0
## containing the fix for this seed issue.