Closed frakc closed 5 years ago
Thanks for the detailed bug report. I'll try to look at this in the next 1-3 days.
OK, so the problem wasn't actually CuDNN at all. The problem here is actually numerical underflow in the softmax that is occuring only when gradient normalization is enabled. Specifically, after a couple of iterations we have values into the softmax that differ by about 9 orders of magnitude (1e5 for largest vs. 1e-4 for the rest). This underflows to probabilites of exactly 1.0 and 0.0, which leads to 0 gradients. Gradient normalization does grad/l2(grad) which is 0.0/0.0 = NaN in this case.
Fixed here: https://github.com/deeplearning4j/deeplearning4j/pull/6546
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
training with gradient normalisation http://dl3.joxi.net/drive/2018/10/04/0030/1076/1967156/56/9e7ab7f3d2.jpg training of same model without gradient normalisation http://joxi.ru/v29x444T3JM1am
version
model