Making use of functional hooks

pfistfl commented 4 years ago

Creating this issue, so we can discuss whether this makes sense.

pytorch, keras etc. make ample use of functional hooks/callbacks.

The same principle is used in Keras as Callbacks. In keras you can for example define a function that is run after / before every batch / epoch.

I think this would nicely fit into mlr3tuning.

Possible use-cases:

After every iteration evaluate the loss on the outer test set (for visualization).
Logging evaluated points to a database
Evaluating predict-step hyperpars i.e. nrounds* < nrounds model was trained with.
Modify a hyperparam of the tuner when a certain criterion is met (for infillCritCB):
- Increase cb.lambda after $n$ iterations.
- Decrease cb.lambda when validation error stagnates
Weave in random evaluations

This could also perform as a Terminator, as such a hook could basically just set some termination flag.

berndbischl commented 4 years ago

in generel yes, we can do this. would it be enough to have this after batch_eval? and i would like to keep this separate from terminators

pfistfl commented 4 years ago

Really depends on which use-cases we want to cover. I guess we should collect a number of situations where we would want this

@mb706

berndbischl commented 4 years ago

Really depends on which use-cases we want to cover.

please outline 1-3 most important ones here. you kinda started in the OP already with that

be-marc commented 3 years ago

@pfistfl https://github.com/mlr-org/bbotk/pull/110 This might cover some of your use-cases. Suggestions are welcome.

pfistfl commented 3 years ago

I answered in the PR.

be-marc commented 1 year ago

Callbacks are implemented in bbotk and mlr3tuning now.

mlr-org / mlr3tuning