Closed RaphaelS1 closed 3 years ago
It works with
at <- AutoTuner$new(
learner = l,
resampling = cv,
measure = MeasureClassifTest$new(),
search_space = ps,
terminator = trm("evals", n_evals = 2),
tuner = tuner,
store_tuning_instance = TRUE,
store_models = TRUE
)
However, this might be inconvenient because it will blow up the result unless you set store_benchmark_result = FALSE
.
Unfortunately we don't allow to have store_models = TRUE
and store_benchmark_result = FALSE
:
https://github.com/mlr-org/mlr3tuning/blob/a8c6bf7eaaae24a5812725164ce3553eaac5c7d1/R/ObjectiveTuning.R#L57-L59
Could we relax that check and document it accordingly?
So, only if keeping all models is not an option, but you need store_benchmark_result = TRUE
we have to implement something smart here:
https://github.com/mlr-org/mlr3tuning/blob/a8c6bf7eaaae24a5812725164ce3553eaac5c7d1/R/ObjectiveTuning.R#L82-L83
But I would say this case can be neglected?
Is there a possibility to store a model up until the next iteration? So the model is kept and benchmarking is run and then the model is removed at the start of the new configuration to be tested
This would happen if you set store_models = TRUE
and store_benchmark_result = FALSE
which currently is not allowed.
Got it. Well then I hope this issue serves as a good use case to show why this may be useful to allow.
We should allow store_models = TRUE
and store_benchmark_result = FALSE
but document that the models are only stored for one iteration and not accessible after the tuning. I am on it.
Closed by 1d23f04b89ae9f88908c1430bb0ede2461127c94
Sorry if this isn't a bug but is expected behaviour.
The example below demonstrates that measures with
requires_learner
are passed the untrained learner in tuning.Created on 2021-01-14 by the reprex package (v0.3.0)