mlr-org / mlr3pipelines

Dataflow Programming for Machine Learning in R
https://mlr3pipelines.mlr-org.com/
GNU Lesser General Public License v3.0
141 stars 25 forks source link

Allow for constructing `GraphLearner` in a trained state #667

Open hrvg opened 2 years ago

hrvg commented 2 years ago

What is the reason for enforcing the construction of a GraphLearner in an untrained state as mentioned in the docs with graph$state = NULL here?

There might be something that I am missing here (this maybe?), but it might be useful to make untrained the default yet allow for constructing trained GraphLearner.,l

My use case is the following. A compute-heavy benchmark with nested resampling across multiple learners was run. Posterior to training, some learners have to be combined in an ensemble. To do so without having to retrain any learners:

  1. collect a list of the desired, trained, tuned learners from the BenchmarkResult
  2. create a list of Graph from each Learner and copy each Learner state to their Graph
  3. pass the list of Graph to mlr3pipelines::gunion()
  4. pass the result to po("classifavg")/po("regravg").

From the resulting Graph, it is possible to use predict. However, predict_newdata only works with a GraphLearner. My current workaround is to hack a Task with desired newdata using Task$filter and Task$rbind. This seemed a simpler approach than hacking the status of the GraphLearner. However, both approaches are hacky and a more streamlined construction would be appreciated to build ensemble from trained models into a proper learner object.

mb706 commented 2 years ago

It could be a bit difficult to have GraphLearner accept a trained Graph and just create a trained Learner from that, since a trained Learner doesn't only contain the trained model state, but also meta-information about the task that was used to train it, which is sometimes necessary for prediction as well. The Graph is OTOH supposed to be a general purpose object, so it doesn't keep track of what it was trained with (since that could be anything).

Something that might work for your case is

gstate <- graph$state
gl <- GraphLearner$new(graph)
gl$state <- list(model = gstate)

The resulting gl is now a somewhat trained Learner that can be used for prediction on tasks, but not for data.frames ($predict() seems to work, $predict_newdata() doesn't work). If necessary, you could get the remaining parts of your individual Learners' $state, change the model-part (inserting the new graph's state), and assign that to gl$state.

Does this help your particular case? I will have to think if there is some nicer way to make this work in general.

hrvg commented 2 years ago

This definitely helps; thanks a lot!

While $predict_newdata() does not work, passing the task as an argument makes it work: gl$predict_newdata(newdata, task). gl$predict() also has better behavior by returning a Prediction instead of list of Prediction.

mb706 commented 3 months ago

Internal question: does it make sense to create a Learner that does not have the training task attached?

mb706 commented 3 months ago

apparently predict_newdata would not work, but the rest might