Wrong Identification of Multi-Class Classification in Metrics Calculation

EFS-OpenSource / calibration-framework

The net:cal calibration framework is a Python 3 library for measuring and mitigating miscalibration of uncertainty estimates, e.g., by a neural network.

https://efs-opensource.github.io/calibration-framework/

Apache License 2.0

336 stars 42 forks source link

Wrong Identification of Multi-Class Classification in Metrics Calculation #38

Closed fabiankueppers closed 1 year ago

fabiankueppers commented 1 year ago

Setup:

Using metrics package for classification (e.g., netcal.metrics.ECE)
Binary classification (number of distinct ground-truth labels: 2)
Input array with shape (n, 2) with n samples and confidence scores for the negative/positive classes, respectively

In this scenario, the metric erroneously identifies the input as mulit-class, although the input is binary. This results in an error.