It would give users a lot of flexibility if we would provide some hooks to evaluate additional code during resample() and benchmark() (and maybe also batchmark(). One important use-case is to offer users a replacement for the holdout task, which we plan to remove soon. We have already made callbacks available in mlr3tuning, mlr3torch and bbotk etc..
Besides giving users the possibility to define their own callbacks, it would be great if we could also offer predefined callbacks for the most common things a user might want to do. Further, it would be great if we provided convenience functions that enable users to define custom callbacks without knowing too much about R6. mlr3tuning and mlr3torch also offer something like this.
Besides evaluating custom code, it should also be possible for these callbacks to store some information in the Result object, e.g. what is returned by a $get_state() method.
It would give users a lot of flexibility if we would provide some hooks to evaluate additional code during
resample()andbenchmark()(and maybe alsobatchmark(). One important use-case is to offer users a replacement for the holdout task, which we plan to remove soon. We have already made callbacks available inmlr3tuning,mlr3torchandbbotketc.. Besides giving users the possibility to define their own callbacks, it would be great if we could also offer predefined callbacks for the most common things a user might want to do. Further, it would be great if we provided convenience functions that enable users to define custom callbacks without knowing too much about R6.mlr3tuningandmlr3torchalso offer something like this. Besides evaluating custom code, it should also be possible for these callbacks to store some information in the Result object, e.g. what is returned by a$get_state()method.