`Perf` not reproducible

[evaham1]

copied from: https://mixomics-users.discourse.group/t/perf-step-gives-inconsistent-results/1691

Hi, I’m trying to work through the steps of building the model for a block.plsda analysis to run DIABLO. I have run block.plsda() followed by perf() but the results keep changing. Any suggestions on what I might be missing to make this step give reproducible results?

I’ve run pls() on each of my X list to determine the design and using that design matrix in the block.plsda().

diablo.MD ← block.plsda(X, Y, ncomp = 3, design = design) perf.diablo.MD = perf(diablo.MD, validation = ‘Mfold’, folds = 6, nrepeat = 10) plot(perf.diablo.MD) perf.diablo.MD$choice.ncomp$WeightedVote

---

🐞 Describe the bug:

Despite setting a seed, perf returns different results when run twice.

---

🔍 reprex results from reproducible example including sessioninfo():

data(liver.toxicity)
X <- liver.toxicity$gene
Y <- liver.toxicity$clinic
liver.pls <- pls(X, Y, ncomp = 5)
set.seed(45)
liver.val.1 <- perf(liver.pls, validation = "Mfold", folds = 5)
liver.val.2 <- perf(liver.pls, validation = "Mfold", folds = 5)
identical(liver.val.1$measures$MSEP$values, liver.val.2$measures$MSEP$values)

---

🤔 Expected behavior:

With seed set, running perf twice with same params on same data should give exactly the same result. This is important for analysis reproducibility and for unit testing.

---

💡 Possible solution:

Seed may not be passed correctly to the perf function

[evaham1]

same issue seen for tune function:

data(multidrug)
X <- multidrug$ABC.trans
set.seed(8589) # for reproducibility with this case study, remove otherwise
test.keepX <- c(seq(5, 25, 5)) # set the number of variable values to be tested

tune.spca.res <- tune.spca(X, ncomp = 3, # generate the first 3 components
                           nrepeat = 5, # repeat the cross-validation 5 times
                           folds = 3, # use 3 folds for the cross-validation
                           test.keepX = test.keepX)
tune.spca.res.2 <- tune.spca(X, ncomp = 3, # generate the first 3 components
                           nrepeat = 5, # repeat the cross-validation 5 times
                           folds = 3, # use 3 folds for the cross-validation
                           test.keepX = test.keepX)

identical(tune.spca.res$cor.comp$comp1$cor.mean, tune.spca.res.2$cor.comp$comp1$cor.mean)

From case study - http://mixomics.org/case-studies/spca-multidrug-case-study/ Again, it says here to set seed for reproducibility but that doesn't seem to work