Closed LynnHo closed 4 years ago
I experienced the same outcome in my implementation when I added learning rate decay every 25k iterations. Whenever the learning rate decreases the loss of G goes up, the real loss of D goes up, the fake loss of D goes down (- loss G) but the overall loss of D continues to increase as normal.
Any idea what could cause this @igul222 @martinarjovsky ? Is there some correct way of doing this learning rate decay? In the paper, it is mentioned that you implemented learning rate decay for the ResNet-101.
Did you manage to find a fix @LynnHo ?
@crisbodnar I have found recently in my work that layer normalization may cause this phenomenon, and instance normalization may help.
@igul222
I have implemented wgan-gp by myself, In my application, G loss sometimes increases much suddenly![image](https://user-images.githubusercontent.com/16457298/32814826-985c9b30-c9eb-11e7-851e-1a0f0db6b64f.png)
But D loss is still stable![image](https://user-images.githubusercontent.com/16457298/32814839-ae45c660-c9eb-11e7-99a5-31310ed68c2d.png)
And the gradient penalty is also stable![image](https://user-images.githubusercontent.com/16457298/32814848-c020a9ea-c9eb-11e7-8a8f-c0b87e8e068e.png)
Any idea? thanks!