Closed gshpychka closed 5 years ago
I am not sure that it has to do with some low threshold.
Training goes something like this:
1/10000 [..............................] - ETA: 26:00 - loss: 0.4511 - precision: 0.3333 - recall: 0.3333 - f1_score: 0.3333
2/10000 [..............................] - ETA: 25:25 - loss: 0.4070 - precision: 0.4167 - recall: 0.3095 - f1_score: 0.3485
3/10000 [..............................] - ETA: 25:23 - loss: 0.4379 - precision: 0.2778 - recall: 0.2063 - f1_score: nan
4/10000 [..............................] - ETA: 25:32 - loss: 0.4203 - precision: 0.2440 - recall: 0.4048 - f1_score: nan
And then it's 'nan's all the way down. Even if precision and recall go up to relatively decent numbers.
What kind of information would help you narrow this down?
@gshpychka, thank you a lot for posting the issue, this looks very strange. I wasn't being able to look into this issue last week, hopefully I'll figure out why is this happening on this one.
Heres my two cents:
100/17172 [..............................] - ETA: 7:09 - loss: 0.0527 - acc: 0.9700 - precision: 0.8000 - recall: 0.8750 - f1_score: 0.8036
150/17172 [..............................] - ETA: 7:08 - loss: 0.0462 - acc: 0.9733 - precision: 0.8667 - recall: 0.8056 - f1_score: 0.8024
200/17172 [..............................] - ETA: 7:07 - loss: 0.0398 - acc: 0.9800 - precision: 0.9000 - recall: 0.8542 - f1_score: 0.8518
250/17172 [..............................] - ETA: 7:07 - loss: 0.0359 - acc: 0.9840 - precision: 0.9200 - recall: 0.8833 - f1_score: 0.8814
300/17172 [..............................] - ETA: 7:06 - loss: 0.0593 - acc: 0.9733 - precision: 0.8778 - recall: 0.8472 - f1_score: 0.8456
350/17172 [..............................] - ETA: 7:06 - loss: 0.0539 - acc: 0.9771 - precision: 0.8952 - recall: 0.8690 - f1_score: 0.8677
400/17172 [..............................] - ETA: 7:05 - loss: 0.0607 - acc: 0.9750 - precision: 0.9083 - recall: 0.8354 - f1_score: 0.8530
450/17172 [..............................] - ETA: 7:03 - loss: 0.0602 - acc: 0.9733 - precision: 0.9185 - recall: 0.8093 - f1_score: 0.8415
500/17172 [..............................] - ETA: 7:02 - loss: 0.0570 - acc: 0.9740 - precision: 0.8267 - recall: 0.7283 - f1_score: nan
550/17172 [..............................] - ETA: 7:00 - loss: 0.0579 - acc: 0.9727 - precision: 0.7970 - recall: 0.7076 - f1_score: nan
600/17172 [>.............................] - ETA: 6:59 - loss: 0.0565 - acc: 0.9750 - precision: 0.8139 - recall: 0.7319 - f1_score: nan
650/17172 [>.............................] - ETA: 6:58 - loss: 0.0531 - acc: 0.9769 - precision: 0.8282 - recall: 0.7526 - f1_score: nan
700/17172 [>.............................] - ETA: 6:56 - loss: 0.0535 - acc: 0.9771 - precision: 0.8405 - recall: 0.7583 - f1_score: nan
750/17172 [>.............................] - ETA: 6:55 - loss: 0.0510 - acc: 0.9787 - precision: 0.8511 - recall: 0.7744 - f1_score: nan
800/17172 [>.............................] - ETA: 6:54 - loss: 0.0503 - acc: 0.9800 - precision: 0.8604 - recall: 0.7885 - f1_score: nan
850/17172 [>.............................] - ETA: 6:53 - loss: 0.0505 - acc: 0.9800 - precision: 0.8569 - recall: 0.8010 - f1_score: nan
Not sure if its connected to P/R values or epoch numer (i.e. it gets overwritten by something/nan) or something else.
I don't actually think that it has to do with the values of P and R. Seems like it disappears after some number of iterations.
See above - it displayed fine at much lower values of P and R, before disappearing.
@gshpychka just updated my comment as in later batches, with P/R > 0.8, nan was calculated as f1 score.
^ In fact, the code that keras used to use is much simpler. Might want to just copy that over.
@gshpychka, @deakkon could you, please, verify that the issue is gone in version 0.0.4
?
Closing the issue, it is resolved in 0.0.5
version.
At some low threshold of precision and recall, f1_score becomes nan. I'm getting outputs like this:
2306/10000 [=====>........................] - ETA: 21:25 - loss: 0.7265 - precision: 0.5095 - recall: 0.4593 - f1_score: nan