Open QiyaoWei opened 2 years ago
Hi Qiyao,
Beware of BatchNorm
; most of the quantities returned by BackPACK are not defined when there's a batchnorm layer in the middle (see e.g. https://github.com/f-dangel/backpack/issues/239).
Easy checks of things that can cause something like this to happen would be calling backward
twice (where the first backward clears the graph, and the second backward
then crashes), and maybe missing a call to backpack.extend(model)
. But this doesn't seem to be the case here.
Only a rough guess looking at the stack, but the error might be specific to BatchNorm.
The error occurs after the computation of the backward pass, during cleanup.
delete_old_quantities = not self.__should_retain_backproped_quantities(module)
. The error 'BatchNorm2d' object has no attribute 'output'
indicates that the extension needed to store additional quantities during the forward pass (the output of the layer) but did not. This is weird; I would expect it to crash much earlier. What extension are you running with batchnorm?
model = get_cls_net()
model = extend(model)
with backpack(BatchGrad()):
model(torch.rand(1,3,32,32)).sum().backward()
The weird thing is, BatchNorm worked with this code when I was trying it on a smaller model, so what I am doing right now is trying to sort out the structural differences between these two models and see if I can find anything useful
is there currently an alternative to BatchNorm?
There are, for example GroupNorm or LayerNorm (see https://pytorch.org/docs/stable/nn.html#normalization-layers). The problem with BatchNorm is that there are no "individual gradient"; it is not possible to isolate the contribution of one sample to the loss because BatchNorm mixes them.
What's the model (get_cls_net
)?
Oh I thought backpack doesn't support GroupNorm
BTW I might have figured out the issue, it goes away when I do add an eval like: extend(model).eval(). Not sure why but I guess that is a fix!
I post the full error below. The MWE is a bit long (currently hundreds of lines) and I am still working on it, but is there any specific direction I should be looking at given this error? It looks like Batchnorm is somehow mixed up in the gradient calculation (judging from the error message)?