lyndonzheng
commented
4 years ago @AnshKhurana The specific training time is hard to be decided for different datasets because of the data distributions. For simple datasets with limited categories (Paris and CelebaHQ), four days are enough to get plausible results. However, for the natural scenes with thousands of categories, if you want to complete various objects, instead of the copying of background, it is a challenge and needs more time. Besides, the GAN still has the mode collapse problem if it is trained for a very long time.
AnshKhurana
As pointed out in other issues, one cannot judge convergence based on losses. However, I am afraid of overfitting and it is quite difficult to judge from looking at the generated images if the model is getting better or if it has stagnated. I wish to evaluate the method as fairly as possible and I am not sure if the model is getting worse with more training. I have a small dataset with around 20K images. When monitoring SSIM over the test set, for instance, I get scores like 0.89, 0.885, 0.905, 0.901 at around 200, 300, 400 and 450 epochs. While there can be slight variations in the figures, it is hard to judge now if the results will get any better or worse.
Trying to match up with the reported 3 days figure for celebA-HQ images, I have been training the model with a batch size of 64 (my images are 128x128) for around 31 hours now (on the 20000 images) so it seems fair to stop training now. Can you give some insight/metric which will allow me to decide if I should stop the training? I am new to GANs and it's difficult for me to make judgements respect to this.
Recent values of loss functions:
(epoch: 450, iters: 140600, time: 0.021) kl_rec: 0.015 kl_g: 0.004 app_rec: 4.295 app_g: 2.359 ad_g: 0.430 img_d: 0.149 ad_rec: 0.392 img_d_rec: 0.100
(epoch: 450, iters: 140700, time: 0.021) kl_rec: 0.015 kl_g: 0.009 app_rec: 4.408 app_g: 2.463 ad_g: 0.880 img_d: 0.134 ad_rec: 0.501 img_d_rec: 0.098
(epoch: 450, iters: 140800, time: 0.021) kl_rec: 0.048 kl_g: 0.026 app_rec: 4.350 app_g: 2.356 ad_g: 0.484 img_d: 0.083 ad_rec: 0.341 img_d_rec: 0.125