search

markus93 / NN_calibration

Calibration of Convolutional Neural Networks
MIT License
157 stars 41 forks source link

why acc_conf.sort(axis=1)? #11

Closed XixiLiu95 closed 3 years ago

XixiLiu95 commented 3 years ago

This might be a mistake... it does not make sense that you sort it...

markus93 commented 3 years ago

Hey!

Could You please refer to the script, where You found this line?

All the best, Markus

XixiLiu95 commented 3 years ago

If you open the NN_calibration/scripts/calibration/Reliability Diagram.ipynb, the section of reliability diagrams as subgraph, the second line of code in function ref_diagram_sub(), you will find it.

markus93 commented 3 years ago

Thanks for referring to the line and asking the question!

The variable "acc_conf" consists of accuracies and the confidences of bins. The sorting is done on the first axis (axis=1), so it sorts the accuracy and confidence bin-wise. It is done so the lower value can be used for the blue bar and the higher value for the gap (red bar). So indeed it is not the best way to visualize confidences and accuracies, as it loses the information, which one is which.

I have improved the code for the reliability diagram [here].(https://github.com/dirichletcal/experiments_dnn/blob/master/scripts/notebooks/Reliability%20Diagrams%20of%20Dirichlet%20(Figure%201%20and%20Supp.%20Figure%2012).ipynb)

XixiLiu95 commented 3 years ago

Hi Markus, Thanks for your response!

Using sort(axis=1) means that you will switch the accuracy-confidence pair if confidence is lower than the accuracy. Moreover, blue columns represent averaged accuracy in the specific confidence interval. Therefore, it is confusing if you switch them. I guess commenting that line of code and changing the name "gap" to "averaged confidence " and "outputs" to "averaged accuracy" is better.

Best, Xixi