Open Xuzzo opened 9 months ago
Hi, as you correctly guessed requires_grad=True
is mandatory for the hooks to be triggered. Not using xx.backward()
was a design choice some time ago and I do not remember the exact reason. I see at least one drawback in that the call to .backward updates the .grad of all tensors, which can interfere in some situations, e.g. in the case of natural gradient you both want to compute the FIM and the average gradients of the loss.
I am afraid I don't have a satisfactory answer. Perhaps you could replace your embedding layer by a Linear layer and transform your input by onehot vectors ?
I have done a few tests and for simple linear and conv models using backward or autograd.grad gives the same results. I think that if differentiating inputs is necessary for some applications it could be made optional in the library. This way one would avoid computational overhead and/or errors like the one I had.
Anyway, I will try to find a workaround (onehot + linear should do the job =) ). Thanks for your answer and for your work.
Hello,
I am calculating the EK-FAC representation on the linear layers of a model with an embedding layer. The inputs are integers which are lookup values and therefore should not be differentiated through. Within the calculation of the kfac blocks in the Jacobian class the inputs are labelled as requiring grad, which gives me
RuntimeError: only Tensors of floating point and complex dtype can require gradients
. As far as I can tell, this is just so that the hooks can be triggered on all the layers, so a simple backward call would be enough.Here is a simple code to replicate the error
Would it be possible to remove the
inputs.requires_grad = True
or make it optional? Thanks a lot for your help and for your very instructive library.