search

igul222 / improved_wgan_training

Code for reproducing experiments in "Improved Training of Wasserstein GANs"
MIT License
2.35k stars 670 forks source link

Cannot reproduce the inception score in the paper #33

Closed cjf00000 closed 7 years ago

cjf00000 commented 7 years ago

Hi,

Excellent work! I am trying to reproduce the 8.42 +- 0.1 inception score in the paper by running gan_cifar_resnet.py for 100000 iterations (which takes about 3 days). Finally I got 8.15 +- 0.08 inception score. Is it because hyperparameters? Do you have any suggested hyperparameter setting to reproduce the experiment?

Thanks!

iter 93999 inception_50k_std 0.108622521162 acc_fake 0.916953146458 time 1.41457112956 dev_cost 1.73758101463 inception_50k 8.30711174011 cost -1.07328498363 acgan 0.00449406914413 acc_real 0.998765647411 wgan -1.07777905464 iter 94999 inception_50k_std 0.0919005274773 acc_fake 0.917750000954 time 1.41755345893 dev_cost 1.74964499474 inception_50k 8.24363517761 cost -1.08436715603 acgan 0.00362730911002 acc_real 0.999234378338 wgan -1.08799433708 iter 95999 inception_50k_std 0.0791481882334 acc_fake 0.918562471867 time 1.42350446486 dev_cost 1.75195538998 inception_50k 8.18746566772 cost -1.09005200863 acgan 0.00292201270349 acc_real 0.999484360218 wgan -1.09297394753 iter 96999 inception_50k_std 0.0875578373671 acc_fake 0.917046904564 time 1.42211779284 dev_cost 1.7690885067 inception_50k 8.21474838257 cost -1.09408867359 acgan 0.0020965943113 acc_real 0.999828100204 wgan -1.09618532658 iter 97999 inception_50k_std 0.0880940034986 acc_fake 0.917890608311 time 1.42067415953 dev_cost 1.7812871933 inception_50k 8.22119998932 cost -1.09451711178 acgan 0.00177419034299 acc_real 0.999937474728 wgan -1.09629142284 iter 98999 inception_50k_std 0.116663098335 acc_fake 0.916062474251 time 1.42149197221 dev_cost 1.7917330265 inception_50k 8.21636009216 cost -1.10520339012 acgan 0.00143045117147 acc_real 0.999953150749 wgan -1.10663378239 iter 99999 inception_50k_std 0.0829349905252 acc_fake 0.917703151703 time 1.41688520479 dev_cost 1.80293142796 inception_50k 8.14605140686 cost -1.10709547997 acgan 0.00111517717596 acc_real 1.0 wgan -1.10821044445

igul222 commented 7 years ago

Unfortunately these results (along with the other GAN Inception score results I'm aware of) have a significant amount of variance between runs. The published numbers were from iter 99999, the first (and only) time we ran that architecture; running it more times might give you a better idea of the distribution of final scores. (Some other papers additionally do early stopping based on Inception score; we opt not to.)

I just re-ran the code from github and I'm getting scores around 8.4, consistent with the reported result:

iter 90999  inception_50k_std   0.0801353901625 acc_fake    0.92029684782   time    0.602583129168  dev_cost    1.91026878357   inception_50k   8.49037361145   cost    -1.03681814671  acgan   0.00762330694124    acc_real    0.99734377861   wgan    -1.04444146156
iter 91999  inception_50k_std   0.0967662632465 acc_fake    0.920796871185  time    0.602234485626  dev_cost    1.93782782555   inception_50k   8.43902873993   cost    -1.04819381237  acgan   0.00605967408046    acc_real    0.997890651226  wgan    -1.05425345898
iter 92999  inception_50k_std   0.0917120203376 acc_fake    0.921718776226  time    0.60294954133   dev_cost    1.95800423622   inception_50k   8.38080024719   cost    -1.04626512527  acgan   0.00479401694611    acc_real    0.998640596867  wgan    -1.05105936527
iter 93999  inception_50k_std   0.0665561929345 acc_fake    0.920562505722  time    0.602898391724  dev_cost    1.95280456543   inception_50k   8.39941883087   cost    -1.05967593193  acgan   0.00390187115408    acc_real    0.99874997139   wgan    -1.06357777119
iter 94999  inception_50k_std   0.0532421544194 acc_fake    0.91821873188   time    0.602506214857  dev_cost    2.0153901577    inception_50k   8.41933727264   cost    -1.06064474583  acgan   0.00334334908985    acc_real    0.999015629292  wgan    -1.06398808956
iter 95999  inception_50k_std   0.0951965004206 acc_fake    0.92062497139   time    0.602440204859  dev_cost    2.01981425285   inception_50k   8.35305118561   cost    -1.06621205807  acgan   0.00243771215901    acc_real    0.999484360218  wgan    -1.06864976883
iter 96999  inception_50k_std   0.0737948045135 acc_fake    0.91759377718   time    0.600785970211  dev_cost    2.05665636063   inception_50k   8.44657325745   cost    -1.07846081257  acgan   0.00219427491538    acc_real    0.999593734741  wgan    -1.08065497875
iter 97999  inception_50k_std   0.09626429528   acc_fake    0.918624997139  time    0.60149699831   dev_cost    2.06579732895   inception_50k   8.36067008972   cost    -1.08249807358  acgan   0.00158038525842    acc_real    0.999859392643  wgan    -1.08407831192
iter 98999  inception_50k_std   0.126528963447  acc_fake    0.920156240463  time    0.602736969948  dev_cost    2.07890248299   inception_50k   8.37297153473   cost    -1.08571732044  acgan   0.00113835313823    acc_real    0.999953150749  wgan    -1.08685576916
iter 99999  inception_50k_std   0.0953019782901 acc_fake    0.918749988079  time    0.601698414564  dev_cost    2.10081338882   inception_50k   8.41027832031   cost    -1.09535109997  acgan   0.000875924131833   acc_real    1.0 wgan    -1.09622704983