search

p-lambda / verified_calibration

Calibration library and code for the paper: Verified Uncertainty Calibration. Ananya Kumar, Percy Liang, Tengyu Ma. NeurIPS 2019 (Spotlight).
MIT License
141 stars 20 forks source link

Unable to obtain calibration error when missing a class #2

Closed mpitropov closed 3 years ago

mpitropov commented 3 years ago

When there are missing classes represented in the ground truth the calibration error cannot be computed.

To reproduce:

>>> cal.get_ece([[0.9,0.1], [0.8,0.2]], [0,0])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/calibration/utils.py", line 196, in get_ece
    binning_scheme=get_equal_prob_bins, mode=mode)
  File "/usr/local/lib/python3.6/dist-packages/calibration/utils.py", line 162, in lower_bound_scaling_ce
    return _get_ce(probs, labels, p, debias, num_bins, binning_scheme, mode=mode)
  File "/usr/local/lib/python3.6/dist-packages/calibration/utils.py", line 232, in _get_ce
    raise ValueError('labels should be between 0 and num_classes - 1.')
ValueError: labels should be between 0 and num_classes - 1.
AnanyaKumar commented 3 years ago

Thanks for pointing this out - should be fixed in the latest commit. Does it work for you now?

mpitropov commented 3 years ago

Yes, thank you for the fix!