search

clvrai / ACGAN-PyTorch

MIT License
265 stars 62 forks source link

Unable to get inception score for cifar-10 when compared to the original paper #7

Open euwern opened 6 years ago

euwern commented 6 years ago

First of all, I would like to thank you for sharing your implementation on github. After running my experiment on cifar-10 and computing the inception score, I discovered that the inception score computed is different from score reported on the original paper.

I get an inception score of 5 for 100 iterations, 5.03 for 200 iterations. 5.24 for 500 iteration. My inception score is calculated using the following tensorflow script. https://github.com/dashayushman/TAC-GAN/blob/master/inception_score.py The reported number for ACGAN paper is 8.25.

The inception score calculation is correct because I can get the same score (11.24), for all 50,000 training images. (as mentioned in Improved Techniques for Training GANs https://arxiv.org/pdf/1606.03498.pdf)

Can you shed some light as why the results differ so much? (5~ vs 8~)

yashbelhe commented 6 years ago

Hey euwren, were you able to find a fix to reproduce the correct inception score on cifar10?

Arrcil commented 5 years ago

euwern， do you deal with the problem??

euwern commented 5 years ago

I am unable to reproduce the correct inception score.

ilyakava commented 5 years ago

@euwern I can't get to 8.25 either. Did you ever do the hyper parameter search they mention in the paper? i.e. "Performing a grid search across 27 hyperparameter configurations" where those were: "generator_learning_rate, discriminator_learning_rate, and discriminator_activation_noise_standard_deviation, for which we used values of [0, 0.0001, 0.0002], [0, 0.0001, 0.0002] and [0, 0.1, 0.2]." In the code here the first two are set to 0.0002 and I think the last is 0 since I don't see any noise added to activations in the code. Let me know. I can't get above 5 IS using this code with the openai tensorflow IS code with achieves 11.23 on the real CIFAR10 data.

ilyakava commented 5 years ago

p.s. @euwern Actually after ensuring the GAN produced image is between 0,1 and then saving, I can get the IS to 6.2. Still quite a bit below the reported number though. But in line with the non-conditional GAN IS for such a network reported elsewhere see table 2