Following our discussion about the 4th (non disease) class, I removed the 4th class and re-ran the experiments overnight. Here are the updated metrics:
For reference, here are the metrics of the same models on 4 classes:
Notes:
The relative performance of the models (w.r.t to each other) is similar, but the absolute BiasedMAE is quite a bit higher(/worse) on 3 classes compared to 4 classes
If we normalize the BiasedMAE by the MAE of a random class-weighted classifier, the metrics are less far apart but still different (~0.38 on 3 classes vs. ~0.33 on 4 classes)
When cherry picking the best performing model, we see:
Following our discussion about the 4th (non disease) class, I removed the 4th class and re-ran the experiments overnight. Here are the updated metrics:
For reference, here are the metrics of the same models on 4 classes:
Notes:
0.38on 3 classes vs. ~0.33on 4 classes)biased_roc_auc = 0.917167&biased_report_F1:class=2=0.860759biased_roc_auc = 0.937691&biased_report_F1:class=2=0.833889