search

NVlabs / stylegan2-ada

StyleGAN2 with adaptive discriminator augmentation (ADA) - Official TensorFlow implementation
https://arxiv.org/abs/2006.06676
Other
1.8k stars 498 forks source link

Sorry, but I suspect your FID is fake #100

Open johnzlj opened 3 years ago

johnzlj commented 3 years ago

I used your provided pretrained model to create 50000 cifar images, and then I used the public TTUR project[1] to calculate the FID. Finally I get FID=66, Instead of the 5.33 you claimed in your paper.

Can you give me an explanation?

[1] TTUR: https://github.com/bioinf-jku/TTUR

balthazar commented 3 years ago

Dupe of #99

Ranazzi commented 1 year ago

Are you using the same sample size? bear in mind that FID is a biased metric

mxochicale commented 1 year ago

@Ranazzi, I would be interested to read more and have better understanding on why FID is a biased metric. Maybe you would like to add further references. Thanks

Ranazzi commented 1 year ago

@mxochicale You can find some content in arxiv.org/abs/1801.01401 and arxiv.org/abs/1911.07023. Feel free to contact me if you need some more content about GANs.

Regards.

mxochicale commented 1 year ago

Thanks @Ranazzi , I did use FID from my previous work in Ultrasound fetal brain imaging and looking into it using diffusion models but will check out shared papers and reach out if I have questions. It would also great that you provide alternatives methods to help @johnzlj and others on quantifying image synthesis.

Ranazzi commented 1 year ago

@mxochicale Some tips I recommend: besides FID, use KID (Kernel Inception Distance); use a sufficiently large sample to compute FID (In my personal experience, I always use at least 20k samples); when FID is used to compare different settings (architectures, losses, etc.) always use the same size during the analysis.

johnzlj commented 1 year ago

oh, sorry, I think maybe I made a mistake. Feel so sorry for interruptions.

john @.***

 

------------------ Original ------------------ From: "NVlabs/stylegan2-ada" @.>; Date: Sat, Jan 28, 2023 12:41 PM @.>; @.**@.>; Subject: Re: [NVlabs/stylegan2-ada] Sorry, but I suspect your FID is fake (#100)

@mxochicale Some tips I recommend: besides FID, use KID (Kernel Inception Distance); use a sufficiently large sample to compute FID (In my personal experience, I always use at least 20k samples); when FID is used to compare different settings (architectures, losses, etc.) always use the same size during the analysis.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

ThanosM97 commented 1 year ago

Hi @johnzlj,

Did you manage to fix your mistake? If so, could you please tell me what was the problem because I am facing the same issue?

Thanks.