search

EFS-OpenSource / calibration-framework

The net:cal calibration framework is a Python 3 library for measuring and mitigating miscalibration of uncertainty estimates, e.g., by a neural network.
https://efs-opensource.github.io/calibration-framework/
Apache License 2.0
326 stars 40 forks source link

Bug in Reliability Diagram #52

Open cdiazbas opened 6 months ago

cdiazbas commented 6 months ago

The ReliabilityDiagram creates bins with values which are not in the input. Code to reproduce it: 

import numpy as np
import matplotlib.pyplot as plt
from netcal.presentation import ReliabilityDiagram
# Generate true and predicts values
y_true = np.random.randint(0, 2, 100).astype(np.float32)
# Generate perfect predictions:
y_pred = y_true.copy() 
n_bins = 10
diagram = ReliabilityDiagram(n_bins)
_ = diagram.plot(y_pred, y_true)

wrong_output

opsavi commented 4 months ago

I experienced the same issue. A quick fix is to just comment line 291 and 292 in ReliabilityDiagram.py https://github.com/EFS-OpenSource/calibration-framework/blob/82e65dc225c7b4443dcd748836ed6b302af12fd2/netcal/presentation/ReliabilityDiagram.py#L291 https://github.com/EFS-OpenSource/calibration-framework/blob/82e65dc225c7b4443dcd748836ed6b302af12fd2/netcal/presentation/ReliabilityDiagram.py#L292