search

FlorianWilhelm / snalu.pytorch

Implementation of Signed NALU & NAC for all real numbers in PyTorch.
MIT License
24 stars 1 forks source link

Very neat #1

Open emrul opened 5 years ago

emrul commented 5 years ago

This is very cool. However, I wanted to report that under some scenarios I have observed NaNs being reported for loss after some epochs. I verified the input data does not have any NaNs and I also checked that the problem does not occur when using regular NALU cells (only affects SNALU).

I will investigate more next week to figure out what causes it.

emrul commented 5 years ago

Exploding gradients problem - should have checked that before opening an issue! Apologies.

FlorianWilhelm commented 5 years ago

Hi @emrul, thanks for letting me know. I also experienced that sometimes it's not working, more in like 10-20% of all cases. That's the downside of Deep Learning in general I guess, it really varies a lot with different random seeds, even more so in fields like Deep Reinforcement Learning.

emrul commented 5 years ago

Thanks @Florian - I think I found a solution though that doesn’t appear to affect the training negatively. Give me a day or two to be satisfied with it and I’ll update a solution here.