Determine AUROC metric

[florianschwanz]

Determine AUROC as quality metric. Follow these tutorials

• https://towardsdatascience.com/understanding-auc-roc-curve-68b2303cc9c5
• https://developers.google.com/machine-learning/crash-course/classification/roc-and-auc
• https://developers.google.com/machine-learning/crash-course/classification/thresholding?hl=en
• https://www.datascienceblog.net/post/machine-learning/performance-measures-multi-class-problems/
• https://stats.stackexchange.com/questions/2151/how-to-plot-roc-curves-in-multiclass-classification

[florianschwanz]

• it makes no sense to calculate AUROC since
  • it is not a binary classifier
  • there is no threshold that can be used
• potentially a ROC curve could be built
  • per class, see https://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html#plot-roc-curves-for-the-multiclass-problem
  • using a threshold value below which no prediction will be made, e.g.
  • with 8 classes the base rate is 12.5%
  • if the highest value of the classifier is below a given threshold, unknown is returned