Problems due to accessing autotuners archive if trained via Graphlearner

[henrifnk]

If I create an autotuner, wrap it in a graphlearner and train the graphlearner , i am not able to access the autotuners archive in the usual manner by at$archive()....

A simple example to reconstruct the issue:

set.seed(123456)
at = AutoTuner$new(lrn("regr.rpart"),
  resampling = rsmp("cv", folds = 3L),
  measures = msr("regr.rmse"),
  tune_ps = ParamSet$new(params = list(ParamDbl$new("cp", lower = 0.01, upper = 0.1))),
  terminator = term("evals", n_evals = 1L),
  tuner = tnr("grid_search", resolution = 1L)
)  
gl_at <- GraphLearner$new(at,task_type = "regr")
gl_at$train(tsk("mtcars"))

After training at$archive() throws Error in at$archive() : attempt to apply non-function and

> at$archive
function (unnest = "no") 
self$tuning_instance$archive(unnest)
<environment: 0x000002790553da88>

same applies more or less to piped autotuners...

pipe = po("nop") %>>% at
pipe$train(tsk("mtcars"))

only in this case archive is somehow accessable through

pipe$pipeops$regr.rpart.tuned$state$model$tuning_instance$archive("tune_x")

[be-marc]

You cannot access the archive because the $model field in AutoTuner is empty when using a graphlearner. at$archive() just returns the stored archive in $model. You can access the archive with gl_at$model$regr.rpart.tuned$model$tuning_instance$archive().

What you describe seems to be normal since the graphlearner never writes something to the $model field of the original learner.

> learner = lrn("regr.rpart")
> gl = GraphLearner$new(learner,task_type = "regr")
> gl$train(tsk("mtcars"))
> learner$model
NULL

[berndbischl]

errm that at the very least seems confusing? but it seems like a violation of a contract somewhere?

[berndbischl]

i am pretty sure that this setup that henri posted shoukd work and that we should test against it

[mb706]

GraphLearner creates a clone of whatever it encapsulates, so the original at is not changed when the pipeline gets tuned. This is consistent with our behaviour in many other places where we tend to create clones when we encapsulate something to avoid bad surprises.

[berndbischl]

@henrifnk i think i also saw an conceptual error in your posted example. why are you wrapping an AutoTuner with a GraphLearner? You would do this, right? Create a Graph. Wrapt it with a GraphLearner. Wrap it with an AT?

[berndbischl]

I THOUGHT the issue would be quite fundamental, and that you wanted to report that this wouldn't work.

ps = ParamSet$new(list(
  ParamInt$new("classif.rpart.minsplit", lower = 1, upper = 10)
))

pl = po("removeconstants") %>>% lrn("classif.rpart")
gl = GraphLearner$new(pl)
at = AutoTuner$new(gl,
  resampling = rsmp("holdout"),
  measure = msr("classif.ce"),
  search_space = ps,
  terminator = term("evals", n_evals = 1L),
  tuner = tnr("grid_search", resolution = 1L)
)
at$train(tsk("iris"))

but it does work.

Can you please give some context WHY you are wrapping the objects above as you do?

[henrifnk]

My intetion was simply to have a static part and a tunable part of the learner. Since both, Graphlearner and Autotuner are "learners" i thought it might not make a difference in which order to connect them.

Still the way you set up your example works for me and solves the problem.

The only remaining way this could cause trouble, that i can guess, is if you would want to have mutiple autotuners in a graph for some reason...

[mb706]

This seems to be solved, afaics the code behaves it is expected here.