search

ja-thomas / autoxgboost

autoxgboost - Automatic tuning and fitting of xgboost
Other
121 stars 19 forks source link

Use Brier score for classification #41

Open ja-thomas opened 6 years ago

ja-thomas commented 6 years ago

Might be better as a default compared to missclassification rate especially for tuning to make everything a bit smoother

PhilippPro commented 6 years ago

In my benchmark, it was not always better regarding the MMCE than for example tuning AUC. It had on average the lowest rank but at least for RF it sometimes provided worse results than tuning MMCE directly or MMCE. Important parameter in my opinion.

ja-thomas commented 6 years ago

Thanks for the info. Looks interesting. So, in essence, it still makes sense for the user to choose the measure that should be optimized. The question is still what's the best default measure :) If I look over the average ranks over different measures, the brier score seems to be the best overall, correct?

PhilippPro commented 6 years ago

Yes, for the ranks this is true. But difference to for example logarithmic loss is not big. Probably it would be interesting to benchmark autoxgboost with different target measures. ;)

ja-thomas commented 6 years ago

sounds like a good idea.

@Coorsaa can you add that to the benchmark?