NaN when accuracy reaches 1 with logistic loss

[c-f-h]

Training a binary classification net with the binary_crossentropy loss function, I found that when the fit method has managed to correctly classify all test cases, the error will become 0 and thus the loss function becomes NaN. Any further training steps cause all future calls to predict to return only NaNs as well, which is certainly not intended.

Ideally, the net would stop training once this occurs. At the very least, the case of the loss function becoming NaN should be handled gracefully.

[fchollet]

I can look into this (I assume this can be solved with a fuzz factor). If you have any code and data to repro this, that would be helpful.

[MarkusPfundstein]

I actually encouter the same issue. But not with binary_crossentropy but with categorical crossentropy and with MSE. The network suddenly starts spitting out NaN values and from that point one never stops doing it. I'd like to give you a sample code, but its part of a big project so not an easy thing to do. My intuition tells me, that its maybe because of overflows. Theano is limited to float32. Maybe the weights are overflowing? If you can give any hints on how to debug this issue than I'd gladly dive into it. Cant proceed with training if this isnt solved. I have a quite mature net now, but it needs further training. However, when I load the weights (which I can share if needed), the NaN errors appears after a couple of iterations.

[fchollet]

I believe this was resolved a while ago.

[MarkusPfundstein]

still happens on newest checkout

[jbmorgado]

I'm getting this behaviour just by running the mnist_cnn.py example.