Closed sebffischer closed 1 week ago
What happens if we set the early stopping set to "train".
The train set is added in line 232. XGBoost uses the last set that is added to the watchlist
as the early stopping set. If you use early_stopping_set = "test"
, the train and test set are added to the watchlist.
So you can plot this:
Yes, we could remove train
. I kept it because it was the default for a few years.
What happens here if the user provides a watchlist?
(1) early_stopping_set = "none"
: The user can do early stopping on an arbitrary set (provided with watchlist
).
(2) early_stopping_set != "none"
: The user can record the performance on an arbitrary set.
Thanks for the clarification, maybe we can document that the last set is used for early stopping as otherwise it is not so easy to understand the code.
There is still a problem if the user provides a watchlist and the learner is wrapped in the graphlearner with preprocessing steps. The preprocessing will not be applied to the watchlist and we would do early stopping on out of distribution data which would probably not be what the user wants. This is similar but not identical to the problem that we already have, that the preprocessing steps are currently not applied to the test set. Maybe we should disable the parameter watchlist
for that reason?
I would not disable the watchlist but also document the pipelines issue. I will document both. Thanks!
this is properly implemented now
Because I just reviewed a PR in extralearners (https://github.com/mlr-org/mlr3extralearners/pull/251), which added early stopping to the survival xgboost learner, I have two questions regarding the early stopping implementation:
https://github.com/mlr-org/mlr3learners/blob/cb8162a4c85ef7183b6141d1519ba27c7cba5428/R/LearnerClassifXgboost.R#L232
What happens here if the user provides a watchlist? Should we even allow to specify a watchlist as we properly support early stopping now? Stuff can go wrong, e.g. in graphlearners etc..
https://github.com/mlr-org/mlr3learners/blob/cb8162a4c85ef7183b6141d1519ba27c7cba5428/R/LearnerClassifXgboost.R#L86
What happens if we set the early stopping set to
"train"
. Looking at the code (https://github.com/mlr-org/mlr3learners/blob/cb8162a4c85ef7183b6141d1519ba27c7cba5428/R/LearnerClassifXgboost.R#L235) it does not seem like it is covered. In the extralearners PR I suggested to disable the"train"
option as nobody would use it anyway.In summary my suggestions are:
(1) Disable
"train"
as allowed factor forearly_stopping_set
(2) disable thewatchlist
hyperparameterWhat do you think @be-marc ?