How to make the ECE figure

[zihaolucky]

Hi @gpleiss could you give a simple code on the figure in this style?

Thanks!

[gpleiss]

Sure thing! Give me a day or two to clean up my plotting code and I'll post a gist!

[zihaolucky]

@gpleiss Great!

[zihaolucky]

I'm trying to reproduce the result in my own application, the ECE could be reduced from 0.016 to 0.008 😂It works but the origin ECE is quite small already.

[gpleiss]

Yeah sometimes you get lucky and a model will actually be pretty well calibrated to begin with!

[zihaolucky]

I'm working on NLP problem, and I did see the NLL loss growing up during training.

[zihaolucky]

@gpleiss

By diving into the (bin_lower, bin_upper, prop_in_bin.data[0], accuracy_in_bin.data[0], avg_confidence_in_bin.data[0])), I found the low confidence bin has large error but the ECE is very small as it's a weighted metric.

The model is still not well calibrated, the final ECE plot is very crucial.

[zihaolucky]

@gpleiss How this going?

[gpleiss]

https://gist.github.com/gpleiss/0b17bc4bd118b49050056cfcd5446c71

This is a rough sketch of how to plot the code. You need all the outputs of a model in a tensor (outputs) and the labels.

I haven't run this code, so there's probably a typo in it. But it should give you a rough idea of what to do.

[zihaolucky]

@gpleiss Thanks!

[SophieChang66]

@zihaolucky hi, this is my network output of the code: Before temperature - NLL: 0.075, ECE: 0.013 Optimal temperature: 1.599 After temperature - NLL: 0.062, ECE: 0.010 the original ECE is small, how do you calibrate your network when the ECE is small ?

[gpleiss]

@SophieChang66 if your ECE is small, then it is well calibrated already :)