[BUGLRN] Bugs in learner <surv.mboost>

[CodeYueXiong]

Expected Behaviour

When trying to evaluate the performance of the mboost models based on IBS score, the $aggregate(msr("surv.mboost")) shall work to get the corresponding IBS scores.

Actual Behaviour

The learner didn't work as expected even if I designed the pipeline to get the distrcompositor in the ppl(), which is shown as follows: lrn_aft = ppl( "distrcompositor", lrn("surv.mboost", baselearner = 'bols', family = "weibull"), graph_learner = TRUE) Also, I have implemented a simple reproducible code based on the 'rats' task. Please see the following outputs and errors.

Reprex

Please use the reprex package to provide a minimal reproducible example of the bug. Please add the argument si = TRUE to include your session info.

### "surv.graf" problem
library("mlr3")
library("mlr3proba")
library("mlr3learners")
library("mlr3extralearners")
#> 
#> Attaching package: 'mlr3extralearners'
#> The following objects are masked from 'package:mlr3':
#> 
#>     lrn, lrns
library("mboost")
#> Loading required package: parallel
#> Loading required package: stabs
#> This is mboost 2.9-3. See 'package?mboost' and 'news(package  = "mboost")'
#> for a complete list of changes.
library("mlr3tuning")
#> Loading required package: paradox
library("mlr3pipelines")

task = mlr_tasks$get("rats")
lrn_aft = lrn("surv.mboost", baselearner = "bols", family = "weibull")
lrn_aft = ppl(
  "distrcompositor", 
  lrn_aft, 
  graph_learner = TRUE)

bmr <- benchmark(benchmark_grid(tasks = task, learners = lrn_aft, resampling = rsmp("cv", folds = 3L)), 
                         store_models = TRUE)
#> INFO  [12:41:27.978] [mlr3]  Benchmark with 3 resampling iterations 
#> INFO  [12:41:28.355] [mlr3]  Applying learner 'surv.kaplan.surv.mboost.compose_distr' on task 'rats' (iter 2/3) 
#> INFO  [12:41:32.433] [mlr3]  Applying learner 'surv.kaplan.surv.mboost.compose_distr' on task 'rats' (iter 1/3) 
#> INFO  [12:41:36.173] [mlr3]  Applying learner 'surv.kaplan.surv.mboost.compose_distr' on task 'rats' (iter 3/3) 
#> INFO  [12:41:40.225] [mlr3]  Finished benchmark

score <- bmr$aggregate(msr("surv.graf"))
#> Error in mlr_reflections$learner_predict_types[[learner$task_type]][[learner$predict_type]]: recursive indexing failed at level 2

^{Created on 2021-02-26 by the reprex package (v0.3.0)}

[RaphaelS1]

Hi, thanks this is actually an issue with mlr3proba and I'll open a corresponding issue there. For now can I suggest using

score <- bmr$score(msr("surv.graf"))

And then manually taking the mean over the three folds?

[CodeYueXiong]

Thanks for the quick reply. I have also tried the score <- bmr$score(msr("surv.graf")), however, it's not working either...

Please refer to the following code:

### "surv.graf" problem
library("mlr3")
library("mlr3proba")
library("mlr3learners")
library("mlr3extralearners")
#> 
#> Attaching package: 'mlr3extralearners'
#> The following objects are masked from 'package:mlr3':
#> 
#>     lrn, lrns
library("mboost")
#> Loading required package: parallel
#> Loading required package: stabs
#> This is mboost 2.9-3. See 'package?mboost' and 'news(package  = "mboost")'
#> for a complete list of changes.
library("mlr3tuning")
#> Loading required package: paradox

task = mlr_tasks$get("rats")
lrn_aft = lrn("surv.mboost", baselearner = "bols", family = "weibull")

bmr <- benchmark(benchmark_grid(tasks = task, learners = lrn_aft, resampling = rsmp("cv", folds = 3L)), 
                         store_models = TRUE)
#> INFO  [09:17:46.597] [mlr3]  Benchmark with 3 resampling iterations 
#> INFO  [09:17:47.089] [mlr3]  Applying learner 'surv.mboost' on task 'rats' (iter 2/3) 
#> INFO  [09:17:47.641] [mlr3]  Applying learner 'surv.mboost' on task 'rats' (iter 3/3) 
#> INFO  [09:17:48.047] [mlr3]  Applying learner 'surv.mboost' on task 'rats' (iter 1/3) 
#> INFO  [09:17:48.437] [mlr3]  Finished benchmark

score <- bmr$aggregate(msr("surv.graf"))
#> Error in assertThat(object, inherits(object, "Distribution"), errormsg): NA is not an R6 Distribution object

^{Created on 2021-03-01 by the reprex package (v0.3.0)}

[fabian-s]

Thanks for the quick reply. I have also tried the score <- bmr$score(msr("surv.graf")), however, it's not working either...

@EchoYueXiong : works for me if you run the code in the first reprex you posted and only replace the last line. your second reprex above omits the crucial "distribution compositor" wrapper around the learner.

[CodeYueXiong]

Thank you for the correction. Yes, it works now!

[RaphaelS1]

Thanks @fabian-s . I'm closing this issue but @EchoYueXiong I will fix the underlying problem in https://github.com/mlr-org/mlr3proba/issues/189 as soon as possible