The classif.featureless learner predicts the most common class when response_type = "prob" but it should predict the outcome mean (i.e. the prevalence, or a logistic regression with only an intercept term). This will give the right results for net benefit analysis for example. As it stands, if 0 is the most common binary label, the learner says all observations have a 0% probably of being 1, which is miscalibrated compared to the outcome mean.
I've changed the predicted probabilities for the default method "mode" accordingly and also improved the docs. If I've missed something, please reopen.
Hello,
The classif.featureless learner predicts the most common class when response_type = "prob" but it should predict the outcome mean (i.e. the prevalence, or a logistic regression with only an intercept term). This will give the right results for net benefit analysis for example. As it stands, if 0 is the most common binary label, the learner says all observations have a 0% probably of being 1, which is miscalibrated compared to the outcome mean.
The relevant lines are: https://github.com/mlr-org/mlr3/blob/main/R/LearnerClassifFeatureless.R#L78-L89
I can do a PR if it would help.
Thanks, Chris