Open emrul opened 5 years ago
Exploding gradients problem - should have checked that before opening an issue! Apologies.
Hi @emrul, thanks for letting me know. I also experienced that sometimes it's not working, more in like 10-20% of all cases. That's the downside of Deep Learning in general I guess, it really varies a lot with different random seeds, even more so in fields like Deep Reinforcement Learning.
Thanks @Florian - I think I found a solution though that doesn’t appear to affect the training negatively. Give me a day or two to be satisfied with it and I’ll update a solution here.
This is very cool. However, I wanted to report that under some scenarios I have observed NaNs being reported for loss after some epochs. I verified the input data does not have any NaNs and I also checked that the problem does not occur when using regular NALU cells (only affects SNALU).
I will investigate more next week to figure out what causes it.