search

mlr-org / bbotk

Black-box optimization framework for R.
https://bbotk.mlr-org.com
GNU Lesser General Public License v3.0
20 stars 9 forks source link

Printer fails #207

Closed sebffischer closed 7 months ago

sebffischer commented 1 year ago

See the bottom of the regex

library(mlr3verse)
#> Loading required package: mlr3
task = tsk("mtcars")
resampling = rsmp("cv")
learner_auto = auto_tuner(
  tnr("random_search", batch_size = 10),
  learner = lrn("regr.rpart"),
  resampling = resampling,
  term_evals = 10
)

learner_auto$train(task)
#> INFO  [12:09:42.718] [bbotk] Starting to optimize 0 parameter(s) with '<OptimizerRandomSearch>' and '<TerminatorEvals> [n_evals=10, k=0]'
#> INFO  [12:09:42.752] [bbotk] Evaluating 1 configuration(s)
#> INFO  [12:09:42.780] [mlr3] Running benchmark with 10 resampling iterations
#> INFO  [12:09:42.817] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 1/10)
#> INFO  [12:09:42.840] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 2/10)
#> INFO  [12:09:42.856] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 3/10)
#> INFO  [12:09:42.873] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 4/10)
#> INFO  [12:09:42.890] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 5/10)
#> INFO  [12:09:42.907] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 6/10)
#> INFO  [12:09:42.924] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 7/10)
#> INFO  [12:09:42.940] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 8/10)
#> INFO  [12:09:42.957] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 9/10)
#> INFO  [12:09:42.974] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 10/10)
#> INFO  [12:09:42.993] [mlr3] Finished benchmark
#> INFO  [12:09:43.028] [bbotk] Result of batch 1:
#> INFO  [12:09:43.032] [bbotk]  regr.mse warnings errors runtime_learners                                uhash
#> INFO  [12:09:43.032] [bbotk]  17.80381        0      0            0.067 edf8c5dc-20df-447b-b001-89c45b7c8044
#> INFO  [12:09:43.033] [bbotk] Evaluating 1 configuration(s)
#> INFO  [12:09:43.044] [mlr3] Running benchmark with 10 resampling iterations
#> INFO  [12:09:43.050] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 1/10)
#> INFO  [12:09:43.067] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 2/10)
#> INFO  [12:09:43.084] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 3/10)
#> INFO  [12:09:43.110] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 4/10)
#> INFO  [12:09:43.128] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 5/10)
#> INFO  [12:09:43.148] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 6/10)
#> INFO  [12:09:43.165] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 7/10)
#> INFO  [12:09:43.182] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 8/10)
#> INFO  [12:09:43.200] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 9/10)
#> INFO  [12:09:43.217] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 10/10)
#> INFO  [12:09:43.234] [mlr3] Finished benchmark
#> INFO  [12:09:43.265] [bbotk] Result of batch 2:
#> INFO  [12:09:43.266] [bbotk]  regr.mse warnings errors runtime_learners                                uhash
#> INFO  [12:09:43.266] [bbotk]  17.80381        0      0            0.064 f240d6e7-8259-4f74-8757-307facfbeb10
#> INFO  [12:09:43.268] [bbotk] Evaluating 1 configuration(s)
#> INFO  [12:09:43.279] [mlr3] Running benchmark with 10 resampling iterations
#> INFO  [12:09:43.287] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 1/10)
#> INFO  [12:09:43.304] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 2/10)
#> INFO  [12:09:43.321] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 3/10)
#> INFO  [12:09:43.340] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 4/10)
#> INFO  [12:09:43.357] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 5/10)
#> INFO  [12:09:43.374] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 6/10)
#> INFO  [12:09:43.391] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 7/10)
#> INFO  [12:09:43.408] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 8/10)
#> INFO  [12:09:43.430] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 9/10)
#> INFO  [12:09:43.447] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 10/10)
#> INFO  [12:09:43.464] [mlr3] Finished benchmark
#> INFO  [12:09:43.494] [bbotk] Result of batch 3:
#> INFO  [12:09:43.496] [bbotk]  regr.mse warnings errors runtime_learners                                uhash
#> INFO  [12:09:43.496] [bbotk]  17.80381        0      0            0.062 e205ed20-7a27-4b0e-b9c7-7c612b297de6
#> INFO  [12:09:43.497] [bbotk] Evaluating 1 configuration(s)
#> INFO  [12:09:43.508] [mlr3] Running benchmark with 10 resampling iterations
#> INFO  [12:09:43.513] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 1/10)
#> INFO  [12:09:43.532] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 2/10)
#> INFO  [12:09:43.554] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 3/10)
#> INFO  [12:09:43.572] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 4/10)
#> INFO  [12:09:43.590] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 5/10)
#> INFO  [12:09:43.609] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 6/10)
#> INFO  [12:09:43.627] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 7/10)
#> INFO  [12:09:43.645] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 8/10)
#> INFO  [12:09:43.664] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 9/10)
#> INFO  [12:09:43.681] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 10/10)
#> INFO  [12:09:43.700] [mlr3] Finished benchmark
#> INFO  [12:09:43.731] [bbotk] Result of batch 4:
#> INFO  [12:09:43.733] [bbotk]  regr.mse warnings errors runtime_learners                                uhash
#> INFO  [12:09:43.733] [bbotk]  17.80381        0      0            0.067 97b33f39-e9ec-449e-9cc7-a27a29b9a90c
#> INFO  [12:09:43.735] [bbotk] Evaluating 1 configuration(s)
#> INFO  [12:09:43.751] [mlr3] Running benchmark with 10 resampling iterations
#> INFO  [12:09:43.756] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 1/10)
#> INFO  [12:09:43.773] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 2/10)
#> INFO  [12:09:43.790] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 3/10)
#> INFO  [12:09:43.807] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 4/10)
#> INFO  [12:09:43.825] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 5/10)
#> INFO  [12:09:43.844] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 6/10)
#> INFO  [12:09:43.862] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 7/10)
#> INFO  [12:09:43.879] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 8/10)
#> INFO  [12:09:43.897] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 9/10)
#> INFO  [12:09:43.915] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 10/10)
#> INFO  [12:09:43.932] [mlr3] Finished benchmark
#> INFO  [12:09:43.963] [bbotk] Result of batch 5:
#> INFO  [12:09:43.965] [bbotk]  regr.mse warnings errors runtime_learners                                uhash
#> INFO  [12:09:43.965] [bbotk]  17.80381        0      0            0.064 5c424629-0747-4c80-b413-9601321ed04f
#> INFO  [12:09:43.966] [bbotk] Evaluating 1 configuration(s)
#> INFO  [12:09:43.977] [mlr3] Running benchmark with 10 resampling iterations
#> INFO  [12:09:43.982] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 1/10)
#> INFO  [12:09:44.000] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 2/10)
#> INFO  [12:09:44.019] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 3/10)
#> INFO  [12:09:44.037] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 4/10)
#> INFO  [12:09:44.056] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 5/10)
#> INFO  [12:09:44.078] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 6/10)
#> INFO  [12:09:44.095] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 7/10)
#> INFO  [12:09:44.114] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 8/10)
#> INFO  [12:09:44.132] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 9/10)
#> INFO  [12:09:44.153] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 10/10)
#> INFO  [12:09:44.177] [mlr3] Finished benchmark
#> INFO  [12:09:44.210] [bbotk] Result of batch 6:
#> INFO  [12:09:44.212] [bbotk]  regr.mse warnings errors runtime_learners                                uhash
#> INFO  [12:09:44.212] [bbotk]  17.80381        0      0            0.068 39185c69-f6a0-4179-a083-c2d9e09c1f51
#> INFO  [12:09:44.213] [bbotk] Evaluating 1 configuration(s)
#> INFO  [12:09:44.225] [mlr3] Running benchmark with 10 resampling iterations
#> INFO  [12:09:44.230] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 1/10)
#> INFO  [12:09:44.249] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 2/10)
#> INFO  [12:09:44.267] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 3/10)
#> INFO  [12:09:44.287] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 4/10)
#> INFO  [12:09:44.306] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 5/10)
#> INFO  [12:09:44.326] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 6/10)
#> INFO  [12:09:44.345] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 7/10)
#> INFO  [12:09:44.363] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 8/10)
#> INFO  [12:09:44.381] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 9/10)
#> INFO  [12:09:44.400] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 10/10)
#> INFO  [12:09:44.418] [mlr3] Finished benchmark
#> INFO  [12:09:44.457] [bbotk] Result of batch 7:
#> INFO  [12:09:44.459] [bbotk]  regr.mse warnings errors runtime_learners                                uhash
#> INFO  [12:09:44.459] [bbotk]  17.80381        0      0            0.068 abb0cccd-a43e-4dba-b2b1-a78063b1504b
#> INFO  [12:09:44.460] [bbotk] Evaluating 1 configuration(s)
#> INFO  [12:09:44.472] [mlr3] Running benchmark with 10 resampling iterations
#> INFO  [12:09:44.477] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 1/10)
#> INFO  [12:09:44.495] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 2/10)
#> INFO  [12:09:44.513] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 3/10)
#> INFO  [12:09:44.532] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 4/10)
#> INFO  [12:09:44.552] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 5/10)
#> INFO  [12:09:44.572] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 6/10)
#> INFO  [12:09:44.590] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 7/10)
#> INFO  [12:09:44.608] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 8/10)
#> INFO  [12:09:44.626] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 9/10)
#> INFO  [12:09:44.644] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 10/10)
#> INFO  [12:09:44.662] [mlr3] Finished benchmark
#> INFO  [12:09:44.698] [bbotk] Result of batch 8:
#> INFO  [12:09:44.701] [bbotk]  regr.mse warnings errors runtime_learners                                uhash
#> INFO  [12:09:44.701] [bbotk]  17.80381        0      0            0.075 30293db5-b223-467d-969b-7eed997243d1
#> INFO  [12:09:44.702] [bbotk] Evaluating 1 configuration(s)
#> INFO  [12:09:44.714] [mlr3] Running benchmark with 10 resampling iterations
#> INFO  [12:09:44.720] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 1/10)
#> INFO  [12:09:44.738] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 2/10)
#> INFO  [12:09:44.762] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 3/10)
#> INFO  [12:09:44.780] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 4/10)
#> INFO  [12:09:44.800] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 5/10)
#> INFO  [12:09:44.817] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 6/10)
#> INFO  [12:09:44.834] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 7/10)
#> INFO  [12:09:44.852] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 8/10)
#> INFO  [12:09:44.869] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 9/10)
#> INFO  [12:09:44.886] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 10/10)
#> INFO  [12:09:44.902] [mlr3] Finished benchmark
#> INFO  [12:09:44.936] [bbotk] Result of batch 9:
#> INFO  [12:09:44.937] [bbotk]  regr.mse warnings errors runtime_learners                                uhash
#> INFO  [12:09:44.937] [bbotk]  17.80381        0      0            0.064 9d27e14a-3ef5-4904-a04e-e91893dc5ff8
#> INFO  [12:09:44.939] [bbotk] Evaluating 1 configuration(s)
#> INFO  [12:09:44.950] [mlr3] Running benchmark with 10 resampling iterations
#> INFO  [12:09:44.956] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 1/10)
#> INFO  [12:09:44.973] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 2/10)
#> INFO  [12:09:44.990] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 3/10)
#> INFO  [12:09:45.007] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 4/10)
#> INFO  [12:09:45.024] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 5/10)
#> INFO  [12:09:45.041] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 6/10)
#> INFO  [12:09:45.058] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 7/10)
#> INFO  [12:09:45.081] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 8/10)
#> INFO  [12:09:45.098] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 9/10)
#> INFO  [12:09:45.115] [mlr3] Applying learner 'regr.rpart' on task 'mtcars' (iter 10/10)
#> INFO  [12:09:45.132] [mlr3] Finished benchmark
#> INFO  [12:09:45.163] [bbotk] Result of batch 10:
#> INFO  [12:09:45.164] [bbotk]  regr.mse warnings errors runtime_learners                                uhash
#> INFO  [12:09:45.164] [bbotk]  17.80381        0      0            0.063 78093228-3984-40cc-9c90-d39a2fdd7eb5
#> INFO  [12:09:45.170] [bbotk] Finished optimizing after 10 evaluation(s)
#> INFO  [12:09:45.170] [bbotk] Result:
#> INFO  [12:09:45.171] [bbotk]  learner_param_vals  x_domain regr.mse
#> INFO  [12:09:45.171] [bbotk]           <list[1]> <list[0]> 17.80381

learner_auto$model$tuning_instance
#> <TuningInstanceSingleCrit>
#> * State:  Optimized
#> * Objective: <ObjectiveTuning:regr.rpart_on_mtcars>
#> * Search Space:
#> Error in `[.data.table`(as.data.table(self$search_space), , c("id", "class", : column(s) not found: id, class, lower, upper, nlevels

Created on 2023-05-17 with reprex v2.0.2

be-marc commented 1 year ago

Thanks! Fails with an empty search space. Will fix in bbotk.

be-marc commented 7 months ago

Fix with bbotk 0.7.3