Hi markus, thanks for this interesting repo, really helpful. I can see there are ece and mce metrics in your evaluation code, I was wondering if the classwise-ece(from your Neurips19) is included in this repo. If not, can I ask a question about the Eq.(2) in your paper. Let's we have a probability matrix which records the output, M(samples) by N(number of classes). When compute cw-ece for class 1, we filter the probability by thresholding the first row(for class 1) and assign each sample into a bin, am I right. Thanks.
You got the calculation of cw-ECE right. This is actually an older repository and does not contain cw-ece scripts, the NeurIPS19 repository is located here and more precisely the code is here.
markus93
commented
4 years ago
Hi markus, thanks for this interesting repo, really helpful. I can see there are ece and mce metrics in your evaluation code, I was wondering if the classwise-ece(from your Neurips19) is included in this repo. If not, can I ask a question about the Eq.(2) in your paper. Let's we have a probability matrix which records the output, M(samples) by N(number of classes). When compute cw-ece for class 1, we filter the probability by thresholding the first row(for class 1) and assign each sample into a bin, am I right. Thanks.