search

EFS-OpenSource / calibration-framework

The net:cal calibration framework is a Python 3 library for measuring and mitigating miscalibration of uncertainty estimates, e.g., by a neural network.
https://efs-opensource.github.io/calibration-framework/
Apache License 2.0
336 stars 42 forks source link

Incorrect documentation for ECE usage #26

Closed leihuayi closed 1 year ago

leihuayi commented 2 years ago

In the readme and API reference docs:

from netcal.metrics import ECE

n_bins = 10

ece = ECE(n_bins)
uncalibrated_score = ece.measure(confidences)
calibrated_score = ece.measure(calibrated)

This triggers:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-39-ea3fcf5ef398> in <module>
      4 
      5 ece = ECE(n_bins)
----> 6 uncalibrated_score = ece.measure(confidences)
      7 calibrated_score = ece.measure(calibrated)
      8 print('uncalibrated_score', uncalibrated_score)

TypeError: measure() missing 1 required positional argument: 'y'

The correct documentation is:

from netcal.metrics import ECE

n_bins = 10

ece = ECE(n_bins)
uncalibrated_score = ece.measure(confidences, ground_truth)
calibrated_score = ece.measure(calibrated, ground_truth)
fabiankueppers commented 2 years ago

Hi, thank you very much for your comment! I definetly missed this point ... will be fixed within the next release.

fabiankueppers commented 1 year ago

Already resolved