Implement test case for Highly Imbalanced Multiclass Datasets

oracle / macest

Model Agnostic Confidence Estimator (MACEST) - A Python library for calibrating Machine Learning models' confidence scores

Universal Permissive License v1.0

100 stars 20 forks source link

Open mrowebot opened 1 year ago

mrowebot commented 1 year ago

Feedback on using MACEst included the following:

Doesn't work well for imbalanced datasets and with high concentration of values close to 0 in Multiclass Classification setting
Set of possible multiple classifications {0, 1, ..., 10}
MACEst returned enormous bounds for confidence

Implement a test case for this and apply MACEst, then fix if necessary/provide instructions on set up.