search

mlr-org / mlr3proba

Probabilistic Learning for mlr3
https://mlr3proba.mlr-org.com/
GNU Lesser General Public License v3.0
128 stars 20 forks source link

Running benchmark with 42 resampling iterations #254

Closed JianGuoZhou3 closed 2 years ago

JianGuoZhou3 commented 2 years ago

I have a new error....

## select holdout as the resampling strategy
resampling <- rsmp("cv", folds = 3)

## add KM and CPH
learners <- c(learners, lrns(c("surv.kaplan", "surv.coxph")))
design <- benchmark_grid(tasks, learners, resampling)
bm <- benchmark(design)
INFO  [06:46:15.292] [mlr3] Running benchmark with 42 resampling iterations 
INFO  [06:46:15.430] [mlr3] Applying learner 'encode.scale.surv.deepsurv.tuned' on task 'whas' (iter 2/3) 
INFO  [06:46:15.674] [bbotk] Starting to optimize 5 parameter(s) with '<OptimizerRandomSearch>' and '<TerminatorEvals> [n_evals=2, k=0]' 
INFO  [06:46:15.719] [bbotk] Evaluating 1 configuration(s) 
INFO  [06:46:15.778] [mlr3] Running benchmark with 1 resampling iterations 
INFO  [06:46:15.788] [mlr3] Applying learner 'surv.deepsurv' on task 'whas' (iter 1/1) 
Error in if (grepl("\\.$", v)) v <- paste0(v, "9000"): argument is of length zero
Traceback:

1. benchmark(design)
2. future.apply::future_mapply(workhorse, task = grid$task, learner = grid$learner, 
 .     resampling = grid$resampling, iteration = grid$iteration, 
 .     mode = grid$mode, MoreArgs = list(store_models = store_models, 
 .         lgr_threshold = lgr_threshold, pb = pb), SIMPLIFY = FALSE, 
 .     USE.NAMES = FALSE, future.globals = FALSE, future.scheduling = structure(TRUE, 
 .         ordering = "random"), future.packages = "mlr3", future.seed = TRUE, 
 .     future.stdout = future_stdout())
3. future_xapply(FUN = FUN, nX = nX, chunk_args = dots, MoreArgs = MoreArgs, 
 .     get_chunk = function(X, chunk) lapply(X, FUN = `[`, chunk), 
 .     expr = expr, envir = envir, future.envir = future.envir, 
 .     future.globals = future.globals, future.packages = future.packages, 
 .     future.scheduling = future.scheduling, future.chunk.size = future.chunk.size, 
 .     future.stdout = future.stdout, future.conditions = future.conditions, 
 .     future.seed = future.seed, future.lazy = future.lazy, future.label = future.label, 
 .     fcn_name = fcn_name, args_name = args_name, debug = debug)
4. value(fs)
5. value.list(fs)
6. resolve(y, result = TRUE, stdout = stdout, signal = signal, force = TRUE)
7. resolve.list(y, result = TRUE, stdout = stdout, signal = signal, 
 .     force = TRUE)
8. signalConditionsASAP(obj, resignal = FALSE, pos = ii)
9. signalConditions(obj, exclude = getOption("future.relay.immediate", 
 .     "immediateCondition"), resignal = resignal, ...)
mllg commented 2 years ago

Cannot reproduce without the tasks. My best guess: Some strange column names or factor levels. Does not look like a problem in mlr3, probably in deepsurv?

RaphaelS1 commented 2 years ago

Actually looks like a {param6} error. @jianguozhouzunyimedicaluniversity can you please let me know version numbers of param6, distr6, mlr3proba?

JianGuoZhou3 commented 2 years ago

Hi @RaphaelS1 

sessionInfo() 
R version 4.0.2 (2020-06-22) Platform: x86_64-pc-linux-gnu (64-bit) Running under: CentOS Linux 7 (Core)  Matrix products: default BLAS:   /usr/local/lib64/R/lib/libRblas.so LAPACK: /usr/local/lib64/R/lib/libRlapack.so  locale:  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8      [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_US.UTF-8       LC_NAME=C                   [9] LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C         attached base packages: [1] parallel  stats     graphics  grDevices utils     datasets  methods   base       other attached packages:  [1] randomForestSRC_3.0.1    devtools_2.4.3           usethis_2.1.5            mlr3viz_0.5.7             [5] mlr3learners_0.5.2       mlr3pipelines_0.4.0      mlr3extralearners_0.5.29 ggfortify_0.4.14          [9] ggplot2_3.3.5            survival_3.2-13          mlr3proba_0.4.4.9000     mlr3_0.13.2              [13] pracma_2.3.6             mboost_2.9-5             stabs_0.6-4              edgeR_3.32.1             [17] limma_3.46.0              loaded via a namespace (and not attached):  [1] fs_1.5.2             RColorBrewer_1.1-2   rprojroot_2.0.2      data.tree_1.0.0      tools_4.0.2           [6] backports_1.4.1      utf8_1.2.2           R6_2.5.1             rpart_4.1-15         DBI_1.1.2            [11] colorspace_2.0-2     withr_2.4.3          mlr3misc_0.10.0      prettyunits_1.1.1    tidyselect_1.1.1     [16] gridExtra_2.3        processx_3.5.2       compiler_4.0.2       cli_3.2.0            ooplah_0.2.0         [21] penalized_0.9-51     lgr_0.4.3            desc_1.4.0           scales_1.1.1         checkmate_2.0.0      [26] nnls_1.4             mvtnorm_1.1-3        quadprog_1.5-8       callr_3.7.0          palmerpenguins_0.1.0 [31] stringr_1.4.0        digest_0.6.29        param6_0.2.4         paradox_0.8.0        set6_0.2.4           [36] pkgconfig_2.0.3      htmltools_0.5.2      sessioninfo_1.2.2    parallelly_1.30.0    fastmap_1.1.0        [41] htmlwidgets_1.5.4    rlang_1.0.1          rstudioapi_0.13      visNetwork_2.1.0     generics_0.1.2       [46] jsonlite_1.7.3       dplyr_1.0.8          magrittr_2.0.2       Formula_1.2-4        Matrix_1.4-0         [51] Rcpp_1.0.8           munsell_0.5.0        fansi_1.0.2          partykit_1.2-15      lifecycle_1.0.1      [56] yaml_2.3.4           stringi_1.7.6.9001   inum_1.0-4           brio_1.1.3           pkgbuild_1.3.1       [61] grid_4.0.2           dictionar6_0.1.3     listenv_0.8.0        crayon_1.5.0         lattice_0.20-45      [66] splines_4.0.2        locfit_1.5-9.4       ps_1.6.0             pillar_1.7.0         uuid_1.0-3           [71] pkgload_1.2.4        codetools_0.2-18     glue_1.6.1           remotes_2.4.2        data.table_1.14.2    [76] vctrs_0.3.8          testthat_3.1.2       distr6_1.6.6         gtable_0.3.0         purrr_0.3.4          [81] tidyr_1.2.0          future_1.24.0        assertthat_0.2.1     cachem_1.0.6         libcoin_1.0-9        [86] tibble_3.1.6         memoise_2.0.1        DiagrammeR_1.0.8     globals_0.14.0       ellipsis_0.3.2
--
 
> | >
>
JianGuoZhou3 commented 2 years ago

This already work now!

JianGuoZhou3 commented 2 years ago

R version 4.0.2 (2020-06-22) Platform: x86_64-pc-linux-gnu (64-bit) Running under: CentOS Linux 7 (Core)

Matrix products: default BLAS: /usr/local/lib64/R/lib/libRblas.so LAPACK: /usr/local/lib64/R/lib/libRlapack.so

locale: [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages: [1] parallel stats graphics grDevices utils datasets methods base

other attached packages: [1] randomForestSRC_3.0.1 devtools_2.4.3 usethis_2.1.5 mlr3viz_0.5.7
[5] mlr3learners_0.5.2 mlr3pipelines_0.4.0 mlr3extralearners_0.5.29 ggfortify_0.4.14
[9] ggplot2_3.3.5 survival_3.2-13 mlr3proba_0.4.4.9000 mlr3_0.13.2
[13] pracma_2.3.6 mboost_2.9-5 stabs_0.6-4 edgeR_3.32.1
[17] limma_3.46.0

loaded via a namespace (and not attached): [1] fs_1.5.2 RColorBrewer_1.1-2 rprojroot_2.0.2 data.tree_1.0.0 tools_4.0.2
[6] backports_1.4.1 utf8_1.2.2 R6_2.5.1 rpart_4.1-15 DBI_1.1.2
[11] colorspace_2.0-2 withr_2.4.3 mlr3misc_0.10.0 prettyunits_1.1.1 tidyselect_1.1.1
[16] gridExtra_2.3 processx_3.5.2 compiler_4.0.2 cli_3.2.0 ooplah_0.2.0
[21] penalized_0.9-51 lgr_0.4.3 desc_1.4.0 scales_1.1.1 checkmate_2.0.0
[26] nnls_1.4 mvtnorm_1.1-3 quadprog_1.5-8 callr_3.7.0 palmerpenguins_0.1.0 [31] stringr_1.4.0 digest_0.6.29 param6_0.2.4 paradox_0.8.0 set6_0.2.4
[36] pkgconfig_2.0.3 htmltools_0.5.2 sessioninfo_1.2.2 parallelly_1.30.0 fastmap_1.1.0
[41] htmlwidgets_1.5.4 rlang_1.0.1 rstudioapi_0.13 visNetwork_2.1.0 generics_0.1.2
[46] jsonlite_1.7.3 dplyr_1.0.8 magrittr_2.0.2 Formula_1.2-4 Matrix_1.4-0
[51] Rcpp_1.0.8 munsell_0.5.0 fansi_1.0.2 partykit_1.2-15 lifecycle_1.0.1
[56] yaml_2.3.4 stringi_1.7.6.9001 inum_1.0-4 brio_1.1.3 pkgbuild_1.3.1
[61] grid_4.0.2 dictionar6_0.1.3 listenv_0.8.0 crayon_1.5.0 lattice_0.20-45
[66] splines_4.0.2 locfit_1.5-9.4 ps_1.6.0 pillar_1.7.0 uuid_1.0-3
[71] pkgload_1.2.4 codetools_0.2-18 glue_1.6.1 remotes_2.4.2 data.table_1.14.2
[76] vctrs_0.3.8 testthat_3.1.2 distr6_1.6.6 gtable_0.3.0 purrr_0.3.4
[81] tidyr_1.2.0 future_1.24.0 assertthat_0.2.1 cachem_1.0.6 libcoin_1.0-9
[86] tibble_3.1.6 memoise_2.0.1 DiagrammeR_1.0.8 globals_0.14.0 ellipsis_0.3.2

 

| >

RaphaelS1 commented 2 years ago

@jianguozhouzunyimedicaluniversity this is very confusing, is the issue open or closed, can you please provide a format of text which is easy to ready if open

JianGuoZhou3 commented 2 years ago

Hi @RaphaelS1 sorry for that.

R version 4.0.2 (2020-06-22)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)

Matrix products: default
BLAS: /usr/local/lib64/R/lib/libRblas.so
LAPACK: /usr/local/lib64/R/lib/libRlapack.so

locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] parallel stats graphics grDevices utils datasets methods base

other attached packages:
[1] randomForestSRC_3.0.1 devtools_2.4.3 usethis_2.1.5 mlr3viz_0.5.7
[5] mlr3learners_0.5.2 mlr3pipelines_0.4.0 mlr3extralearners_0.5.29 ggfortify_0.4.14
[9] ggplot2_3.3.5 survival_3.2-13 mlr3proba_0.4.4.9000 mlr3_0.13.2
[13] pracma_2.3.6 mboost_2.9-5 stabs_0.6-4 edgeR_3.32.1
[17] limma_3.46.0

loaded via a namespace (and not attached):
[1] fs_1.5.2 RColorBrewer_1.1-2 rprojroot_2.0.2 data.tree_1.0.0 tools_4.0.2
[6] backports_1.4.1 utf8_1.2.2 R6_2.5.1 rpart_4.1-15 DBI_1.1.2
[11] colorspace_2.0-2 withr_2.4.3 mlr3misc_0.10.0 prettyunits_1.1.1 tidyselect_1.1.1
[16] gridExtra_2.3 processx_3.5.2 compiler_4.0.2 cli_3.2.0 ooplah_0.2.0
[21] penalized_0.9-51 lgr_0.4.3 desc_1.4.0 scales_1.1.1 checkmate_2.0.0
[26] nnls_1.4 mvtnorm_1.1-3 quadprog_1.5-8 callr_3.7.0 palmerpenguins_0.1.0
[31] stringr_1.4.0 digest_0.6.29 param6_0.2.4 paradox_0.8.0 set6_0.2.4
[36] pkgconfig_2.0.3 htmltools_0.5.2 sessioninfo_1.2.2 parallelly_1.30.0 fastmap_1.1.0
[41] htmlwidgets_1.5.4 rlang_1.0.1 rstudioapi_0.13 visNetwork_2.1.0 generics_0.1.2
[46] jsonlite_1.7.3 dplyr_1.0.8 magrittr_2.0.2 Formula_1.2-4 Matrix_1.4-0
[51] Rcpp_1.0.8 munsell_0.5.0 fansi_1.0.2 partykit_1.2-15 lifecycle_1.0.1
[56] yaml_2.3.4 stringi_1.7.6.9001 inum_1.0-4 brio_1.1.3 pkgbuild_1.3.1
[61] grid_4.0.2 dictionar6_0.1.3 listenv_0.8.0 crayon_1.5.0 lattice_0.20-45
[66] splines_4.0.2 locfit_1.5-9.4 ps_1.6.0 pillar_1.7.0 uuid_1.0-3
[71] pkgload_1.2.4 codetools_0.2-18 glue_1.6.1 remotes_2.4.2 data.table_1.14.2
[76] vctrs_0.3.8 testthat_3.1.2 distr6_1.6.6 gtable_0.3.0 purrr_0.3.4
[81] tidyr_1.2.0 future_1.24.0 assertthat_0.2.1 cachem_1.0.6 libcoin_1.0-9
[86] tibble_3.1.6 memoise_2.0.1 DiagrammeR_1.0.8 globals_0.14.0 ellipsis_0.3.2
JianGuoZhou3 commented 2 years ago

I just have another question: how to get the model and coefficient? like you said before:

model = as.data.table(bmr)$learner[[8]]$model
model = bmr$resample_results$resample_result[[1]]$learners[[1]]$model
model = bmr$resample_result(1)$learners[[1]]$model
coef(bmr)

but the result is NULL. image

RaphaelS1 commented 2 years ago

Sorry I don't know what you're showing me. If this is a different issue/question then please close this one and open a new one with a reprex