Strange G_vgg loss curve

[HaozhouPang]

Hello, thank you for posting this great work!

I have retrained the model with a customized dataset, the results look great but the loss curves seem strange to me.

The adversary loss seems ok, I set the weights for D and G to 200 and 300, respectively, and the losses are approaching the equilibrium.

However, the G_vgg loss, which consists of c_loss, s_loss, color_loss, tv_loss, reaches the bottom at around epoch 30 and then starts increasing. I looked at each individual loss among G_vgg_loss, only the s_loss is decreasing over time, and all others starts increasing after epoch 30.

Interestingly, the validation samples from epoch 100 is apparently better than the ones from epoch 30. Does anyone experience the same?

[TachibanaYoshino]

With the iteration of training, the loss of the discriminator is gradually reduced, which is the optimal convergence of its discriminative ability. The generator loss will gradually converge to an equilibrium position following the discriminator loss. In the process of style transfer, the content loss will definitely increase gradually, because the initialized generator generates realistic pictures. Then, with the stylization training, the generated pictures will no longer be so realistic, so the content loss will gradually increase . Regarding the style loss, its decrease indicates that the stylized training is making progress, which is consistent with the training goal. The color reconstruction loss is essentially a part of the content loss. It is the calculation of the pixel-level difference between the generated image and the input photo, but it is processed through different color formats.

[HaozhouPang]

Thanks for the quick response and your clarification makes it crystal clear lol!

b.t.w, I noticed that the color loss and tv_loss are negligible comparing with other losses, (in my training, they are both less than 1 even after multiplied by their weights), is this normal or did I miss something important?

[TachibanaYoshino]

Actually, color reconstruction loss and tv_loss do not need more weights, they are just auxiliary functions. If the color reconstruction loss is too great, the generated image will appear very real. If tv_loss is too large, the resulting image will become very blurry. As shown in the screenshot below (from AnimeGANv3), the tv_loss weight on the left picture is 100, and the tv_loss weight on the right picture is 10. The weight of tv_loss in the third image is 1000. In fact, they still look blurry. The tv_loss is generally used as a regularization term of the objective function to smooth the generated image. Its goal is to punish the difference between adjacent pixels in the horizontal and vertical directions, so that the difference between pixels in the entire picture is smaller and the image will be smoother. It has certain resistance to noise and artifacts, but when the weight of this regularization term is too large, the generated image will become very blurry due to excessive smoothing.

[HaozhouPang]

Got it! Thanks for your explanation and really appreciate your time to collect those amazing demo images. I am closing this issue and looking forward to seeing AnimeGANv3 comes out.