search

NVlabs / edm

Elucidating the Design Space of Diffusion-Based Generative Models (EDM)
Other
1.37k stars 143 forks source link

Can not reproduct the result in Table 2. #16

Open Zyriix opened 11 months ago

Zyriix commented 11 months ago

I run the following training command on 8 v100.

torchrun --standalone --nproc_per_node=8 train.py --outdir=training-runs \ --data=datasets/cifar10-32x32.zip --cond=0 --arch=ddpmpp

This gives me a FID at 2.08169 which is still far from FID in the paper(1.97).

I think this may caused by the random seeds(Random init in the code). Is it possible to share the seed for reprodcuting the result in table2?

Any suggestiones?

Newbeeer commented 11 months ago

I observed the FID of EDM fluctuates largely. Did you report the lowest FID among checkpoints?

Zyriix commented 11 months ago

I observed the FID of EDM fluctuates largely. Did you report the lowest FID among checkpoints?

I found this in some papers that try to retrain Edm. I got the lowest FID(2.04053) at around 16w ckpt. I think the fluctuation is caused by lower ema rate and larger learning rate.

yuanzhi-zhu commented 11 months ago

@Zyriix To calculate FID we have to first generate, saying 50000 samples. The random seed can influence the generation process of these samples hence the FID value I guess

Zyriix commented 11 months ago

@Zyriix To calculate FID we have to first generate, saying 50000 samples. The random seed can influence the generation process of these samples hence the FID value I guess

Most works use the same sample seeds just like this repo: 0~49999, 50000~99999, 100000~149999 and report the lowest one.

Liyuan-Liu commented 9 months ago

--arch=ncsnpp performs much better than VP architecture