search

mlr-org / bbotk

Black-box optimization framework for R.
https://bbotk.mlr-org.com
GNU Lesser General Public License v3.0
20 stars 9 forks source link

The handling of trafos and the various $eval(), $eval_batch(), $eval_dt() things are more difficult than they should be #189

Closed mb706 closed 1 year ago

mb706 commented 2 years ago

We should think about how to make handling of the various $eval_**() functions easier. I had to write a few functions that were wrapping other Objectives or TuningObjectives. Calling $eval_many() from inside the .fun of a ObjectiveRFunDt doesn't straightforwardly work when there are missings or trafos involved. I needed to use Design, but it feels wrong to construct an object when what I want to do is transform one value to another.

Having to re-invent the TunerObjective should also not be necessary when I need to more things during tuning than just benchmarking.

be-marc commented 1 year ago

Yes, missings are a bit tricky with data tables. Missings are NA in a data table but NA can also be a valid parameter value. We decided that the user needs to handle NAs / Missings when using ObjectiveRFunDt. But I'm looking forward to hearing your thoughts.

Having to re-invent the TunerObjective should also not be necessary when I need to more things during tuning than just benchmarking.

Shouldn't that soon be possible with callbacks?

mb706 commented 1 year ago

Turns out, one problem with ObjectiveRFunDt is that, if the search space has a trafo, eval_many is called with the transformed values (xss, list of list of param values). ObjectiveRFunDt then assembles a data.table from that list by calling rbindlist. If the xss is short, e.g. only contains one paramester configuration point, and if a hyperparameter is missing, then the column for that hyperparameter is not going to be there.

I had to resort to this hack to make sure the missing columns are there, since the underlying function relied on the columns all being there and having the right type (and being NA if necessary).

mb706 commented 1 year ago

this rbindlist should be augmented to make sure all required columns are there (and have the right type)

mb706 commented 1 year ago

Also suggestion: ObjectiveRFunMany (?) where the given fun is called with the xss through eval_many

be-marc commented 1 year ago

this rbindlist should be augmented to make sure all required columns are there (and have the right type)

Solved by #198. Improves #143.

be-marc commented 1 year ago

Also suggestion: ObjectiveRFunMany (?) where the given fun is called with the xss through eval_many

199