Closed hyskylord closed 2 years ago
Hi hyskylord, I am curious about why this error happens. There seems to be some root cause elsewhere (e.g. bugs in libriichi library or invalid training data). It would be helpful if you can provide more details on how to reproduce the error.
@wongsingfo I think it is because of special training data but I cannot provide reproducible examples (it only happens around every 100k batch). I don't know why I get mu=NaN sometimes but it is quite reasonable to expect logsig.exp() has some zeros if logsig has some sufficiently small terms (less than -1000 or so)
NaN is caused by some numerical instability issue which is not uncommon in deep learning, especially when AMP is on. I prefer letting the error get raised as is, because sometimes it implies the model is actually malformed (weights become NaN themselves) and some rollback is required.
As for the second term, logsig.exp()
collapsing to 0, it is better to just add some constant eps value (eg. 1e-6) to it to make it always stay positive. In fact such mechanism is already implemented in Mortal v2, but not ported to v1 (this repo) yet.
This PR resolve occasional ValueError when training offline.