autoplot not producing different plots when imputted with different predict_sets

[jpconnel]

Description

predict_sets parameter in 'autoplot' for type = 'roc' seems to change nothing in the resulting plots

Reproducible example

task = tsks("german_credit")
learner = lrns("classif.ranger", predict_type = "prob")
learner$classif.ranger$predict_sets <- c("test", "train")
rsmp_cv5 = rsmp("cv", folds = 5)
set.seed(1234)
res = resample(task = task[[1]], learner = learner[[1]], resampling =rsmp_cv5)
autoplot(res, type = 'roc', predict_sets = c("train", "test"))
autoplot(res, type = 'roc', predict_sets = "train")
autoplot(res, type = 'roc', predict_sets = "test")

[bblodfon]

Hi @jpconnel,

Sorry for the late reply!

Please check the corresponding documentation page => https://mlr3viz.mlr-org.com/reference/autoplot.ResampleResult.html. This only works with 2 features and type = "prediction". type = 'roc' is not different no matter what is the predict_sets.

library(mlr3)
library(mlr3learners)
library(mlr3viz)
task = tsk("german_credit")
task$select(cols = c("savings", "age"))

learner = lrn("classif.ranger", predict_type = "prob", predict_sets = c("test", "train"))
cv = rsmp("cv", folds = 5)
res = resample(task = task, learner = learner, resampling = cv, store_models = TRUE)
#> INFO  [15:03:25.252] [mlr3] Applying learner 'classif.ranger' on task 'german_credit' (iter 1/5)
#> INFO  [15:03:26.722] [mlr3] Applying learner 'classif.ranger' on task 'german_credit' (iter 2/5)
#> INFO  [15:03:26.915] [mlr3] Applying learner 'classif.ranger' on task 'german_credit' (iter 3/5)
#> INFO  [15:03:27.096] [mlr3] Applying learner 'classif.ranger' on task 'german_credit' (iter 4/5)
#> INFO  [15:03:27.281] [mlr3] Applying learner 'classif.ranger' on task 'german_credit' (iter 5/5)

autoplot(res, type = 'prediction', predict_sets = c("train", "test"))

^{Created on 2024-07-02 with reprex v2.0.2}

@be-marc maybe adding predict_sets in type = roc or pr is a good idea?