I trained the generator for one time, and trained the discriminator for five times in a iteration. After training for 20k iterations, I found that, the distance between the fake_loss and real_loss is stable at a fixed value, about 30. Although the d_loss which consists the d_f_loss, d_r_loss, and gradient_penalty_loss is decreasing in a much slow rate, the distance between the fake_loss and real_loss is almost invariable, only the gradient_penalty_loss is decreasing to less than 10. And as the gradient_penalty_loss with the lambda 10 and the ideal the gradient is 1, I think the desired the gradient_penalty_loss is 10. So I think although the d_loss is decreasing, the generator is not updated, as the d_r_loss + d_fake_loss is stable.
How do you like it? Should I stop my training early or train the generator more times?
Here is my part of training log.
Hello, I trained a WGAN-GP on the 3D voxel dataset, and my loss code is:
I trained the generator for one time, and trained the discriminator for five times in a iteration. After training for 20k iterations, I found that, the distance between the fake_loss and real_loss is stable at a fixed value, about 30. Although the d_loss which consists the d_f_loss, d_r_loss, and gradient_penalty_loss is decreasing in a much slow rate, the distance between the fake_loss and real_loss is almost invariable, only the gradient_penalty_loss is decreasing to less than 10. And as the gradient_penalty_loss with the lambda 10 and the ideal the gradient is 1, I think the desired the gradient_penalty_loss is 10. So I think although the d_loss is decreasing, the generator is not updated, as the d_r_loss + d_fake_loss is stable. How do you like it? Should I stop my training early or train the generator more times? Here is my part of training log.