Closed zihaolucky closed 6 years ago
Sure thing! Give me a day or two to clean up my plotting code and I'll post a gist!
@gpleiss Great!
I'm trying to reproduce the result in my own application, the ECE could be reduced from 0.016 to 0.008 😂It works but the origin ECE is quite small already.
Yeah sometimes you get lucky and a model will actually be pretty well calibrated to begin with!
I'm working on NLP problem, and I did see the NLL loss growing up during training.
@gpleiss
By diving into the (bin_lower, bin_upper, prop_in_bin.data[0], accuracy_in_bin.data[0], avg_confidence_in_bin.data[0]))
, I found the low confidence bin has large error but the ECE is very small as it's a weighted metric.
The model is still not well calibrated, the final ECE plot is very crucial.
@gpleiss How this going?
https://gist.github.com/gpleiss/0b17bc4bd118b49050056cfcd5446c71
This is a rough sketch of how to plot the code. You need all the outputs of a model in a tensor (outputs
) and the labels.
I haven't run this code, so there's probably a typo in it. But it should give you a rough idea of what to do.
@gpleiss Thanks!
@zihaolucky hi, this is my network output of the code: Before temperature - NLL: 0.075, ECE: 0.013 Optimal temperature: 1.599 After temperature - NLL: 0.062, ECE: 0.010 the original ECE is small, how do you calibrate your network when the ECE is small ?
@SophieChang66 if your ECE is small, then it is well calibrated already :)
Hi @gpleiss could you give a simple code on the figure in this style?
Thanks!