G loss increases much suddenly?

[LynnHo]

@igul222

I have implemented wgan-gp by myself, In my application, G loss sometimes increases much suddenly

But D loss is still stable

And the gradient penalty is also stable

Any idea？ thanks!

[crisbodnar]

I experienced the same outcome in my implementation when I added learning rate decay every 25k iterations. Whenever the learning rate decreases the loss of G goes up, the real loss of D goes up, the fake loss of D goes down (- loss G) but the overall loss of D continues to increase as normal.

Any idea what could cause this @igul222 @martinarjovsky ? Is there some correct way of doing this learning rate decay? In the paper, it is mentioned that you implemented learning rate decay for the ResNet-101.

Did you manage to find a fix @LynnHo ?

[LynnHo]

@crisbodnar I have found recently in my work that layer normalization may cause this phenomenon, and instance normalization may help.