search

atong01 / conditional-flow-matching

TorchCFM: a Conditional Flow Matching library
https://arxiv.org/abs/2302.00482
MIT License
1.25k stars 101 forks source link

Is it normal that FID affected by the num_gen option? #111

Closed glynpu closed 7 months ago

glynpu commented 7 months ago

Thanks for this great work.

I have reproduced the claimed FID 3.5 on cifar10. But the default FID computation is quite time consuming(about 40 minutes) since it requires generating 50,000 images.

When I reduce the num_gen option to 5,000 in compute_fid.py, the resulted FID increases a lot, e.g. increased from 5.12 to 9.75.

Is this normal? According FID's definition, FID shall not affected by the num_gen.

image
kilianFatras commented 7 months ago

Yes, it is normal. The FID is defined as the squared 2-Wasserstein distance between two approximated Gaussians. We first embed generated and real data within the deepest Inception layer (to compare features rather than pixels) and then, we compute Gaussian statistics of the generated and real samples. The more samples we have to compute the Gaussian statistics, the better the approximation is.

The process takes indeed ~35 min on 1 A100. Increasing the batch size and the number of GPUs can reduce this time accordingly.