search

slds-lmu / paper_2019_multiobjective_rfms

High Dimensional Restrictive Federated Model Selection with multi-objective Bayesian Optimization over shifted distributions
https://github.com/smilesun/tex_2018_intellisys2019_fmoboms
2 stars 0 forks source link

tuning threshold VS tuning AUC #18

Closed smilesun closed 5 years ago

smilesun commented 5 years ago

for classification, the threshold can be tuned by mlr, how to do that? what is the advantage and disadvantage compared to tuning auc? are the two better than tuning mmce?

smilesun commented 5 years ago

maybe tuning brier score is the best, since other measure might have matrix inversion problem during GP

pfistfl commented 5 years ago

Threshold tuning makes sense for measures that require a response not a probabilitiy

By adjusting the threshold we can then, for example obtain higher accurarcies. For measures that require probabilities, tuning the threshold does not make sense.

I would suggest brier score.