Why PipeOpTuneThreshold reduce the classif.ce?

[loveyu3317]

In the post about TuneThreshold, classif.ce was used for tunethreshold.measure. But why in result of benchmark the classif.ce for classif.rpart.tunethreshold was bigger than no_tuning model?

[pfistfl]

Hey, good catch!

This can happen due to various reasons, the most likely one, in this case, being that we train a "cross-validated" learner, so each model only gets to see 2/3 of the training data (in comparison to the original one which gets to see all training data). For smaller datasets, such as the ones in our example, this sometimes leads to poorer results. Other reasons can include just pure randomness/overfitting if the differences between the default threshold (0.5) and the tuned threshold are small.

I have added a different dataset as well as a seed and some more explanations in #108 .