Questions about code

[sweetdream33]

Hi, thank you for publishing the code. (1) I am trying to run your code but I found a strange point. The ECE value varies greatly depending on the type and parameter of the optimizer. I have experimented with LBFGS and Adam adjusting learning rate, max_iter.

So I added this line 'optimizer.zero_grad()' to the original code. Adding this line will fix the ECE value to some extent. Is it right to add this?

           loss = nll_criterion(self.temperature_scale(val_logits), val_labels)
            optimizer.zero_grad()
            loss.backward()

(2) Also, I added this line 'model.eval()' before 'logits_list = []'. If I add this, ECE values ​​are better, do not I need to add it?

I'll wait for an answer.

[kirk86]

@sweetdream33
(1) Indeed I've noticed the same, ECE values vary depending on optimizer and hyperparam choice such as max_iter.

Is it right to add this?

I presume yes, although I'm not expert in pytorch but all the examples I've seen in the training loop before everything else they zero the grads of the model and optimizer.

model.zero_grad()
optimizer.zero_grad()
nll_criterion(...

[dreamflasher]

My impression is that the changes you are seeing are mostly from the np and pytorch randomness. See here: https://github.com/gpleiss/temperature_scaling/issues/16

When I fix the seeds, neither model.eval() nor model.zero_grad() have any impact anymore. optimizer.zero_grad() still makes a difference, though it does not constantly improve.

[yshvrdhn]

https://stats.stackexchange.com/questions/284712/how-does-the-l-bfgs-work/285106 . I think since we are using L-BFGS we should not be calling optimizer.zero_grad() after each minibatch and let it accumulated for several minibatch and than do the update and set the gradients to zero again. that might help improve the temperature.