Closed stanleee5 closed 4 years ago
Yeah I think it's high time we introduce this fix, and fix things upstream when they break.
If people want the original behaviour where gradients are not scaled, you will need to remove any averaging done on the loss as this will automatically apply to the gradients now!
It seems that _gradoutput of torch.autograd.Function saves the gradient of the layer from the front of it. When i scaled CTC loss (like using AMP), i had to scale the graident too.