my simple correction for reproducing good performance

[hyunjunChhoi]

(1) adding model.eval() model.eval() model.set_temperature(valid_loader) (before) Before temperature - NLL: 2.441, ECE: 0.247 Optimal temperature: 1386304.250 After temperature - NLL: 4.605, ECE: 0.580

(after) Before temperature - NLL: 2.249, ECE: 0.234 Optimal temperature: 5.823 After temperature - NLL: 1.572, ECE: 0.271

(2)adding optimizer.zero_grad() def eval(): loss = nll_criterion(self.temperature_scale(logits), labels) optimizer.zero_grad()

then, Before temperature - NLL: 2.249, ECE: 0.234 Optimal temperature: 1.800 After temperature - NLL: 1.460, ECE: 0.139

(3)change max iter 40(default) to 10000

Before temperature - NLL: 2.249, ECE: 0.234 Optimal temperature: 3.035 After temperature - NLL: 1.246, ECE: 0.016

(4)change nll_loss to ece_loss (max iter 10000) def eval():

loss = nll_criterion(self.temperature_scale(logits), labels)

        loss = ece_criterion(self.temperature_scale(logits), labels)
        optimizer.zero_grad()

Before temperature - NLL: 2.249, ECE: 0.234 Optimal temperature: 2.981 After temperature - NLL: 1.247, ECE: 0.011

I think it reproduces similar performance with the original paper

maybe it depends on hyperparameter

any other issues ?? I don't know whether it is the right correction or not

[hyunjunChhoi]

On the test data: I got Test Accuracy of the model on the test images: 66.1 % Test Accuracy of the model on the test images: 66.1 % before temperature - NLL: 2.199, ECE: 0.227 After temperature - NLL: 1.216, ECE: 0.012

[ltong1130ztr]

I think adding model.eval() in the ModelWithTemperature() class definition is a safer way, some flat classifier has batch normalization or dropout layers, if someone loading these type of model without turning it into eval mode, then the output raw logits are corrupted. It would be something like this:

class ModelWithTemperature(nn.Module):
    """
    A thin decorator, which wraps a model with temperature scaling
    model (nn.Module):
        A classification neural network
        NB: Output of the neural network should be the classification logits,
            NOT the softmax (or log softmax)!
    """
    def __init__(self, model):
        super(ModelWithTemperature, self).__init__()
        self.model = model
        self.model.eval()
        self.temperature = nn.Parameter(torch.ones(1) * 1.5)