search

w86763777 / pytorch-ddpm

Unofficial PyTorch implementation of Denoising Diffusion Probabilistic Models
Do What The F*ck You Want To Public License
477 stars 60 forks source link

How to lower FID #22

Closed parkjjoe closed 3 months ago

parkjjoe commented 3 months ago

Thanks for providing the code. But, I think an error occured during evaluation.

Untitled

I downloaded your checkpoint model and tried to evaluate using this code: CUDA_VISIBLE_DEVICES=0 python main.py --eval --num_images 50000 --flagfile ./logs/DDPM_CIFAR10_EPS/flagfile.txt. And the output script was this:

Files already downloaded and verified
Model params: 34.21 M
generating images: 100%|????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 391/391 [7:53:24<00:00, 72.65s/it]
/home/parkjoe/anaconda3/envs/pytorch-ddpm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.????????????????????????????????????????| 391/391 [7:53:24<00:00, 64.47s/it]
  warnings.warn(
/home/parkjoe/anaconda3/envs/pytorch-ddpm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=None`.
  warnings.warn(msg)
generating images: 100%|????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 391/391 [7:51:47<00:00, 72.40s/it]
0/800000 IS:1.174, IS_std:0.001, FID:460.166, IS_EMA:1.176, IS_std_EMA:0.002, FID_EMA:460.171                                                                                                                                                                                                                       
  7%|???????????????????                                                                                                                                                                                                                                   | 59259/800000 [20:24:53<49:49:44,  4.13it/s, loss=0.029]
7%|???????????????????                                                                                                                                                                                                                                  | 59259/800000 [20:24:53<255:11:17,  1.24s/it, loss=0.029]

At this, I have some questions. First, what do those warning messages mean? Is it normal for those messages to be printed out? Second, why does the FID print out as 460.166? Does this relate to the first question? Third, what is the progress bar that appears after the performance indicators?

parkjjoe commented 3 months ago

I reuploaded the checkpoint model and retried to evaluate using this code: CUDA_VISIBLE_DEVICES=3 python main.py --flagfile ./logs/DDPM_CIFAR10_EPS/flagfile.txt --notrain --eval. And the output script was this:

generating images: 100%|????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 391/391 [7:52:59<00:00, 72.58s/it]
/home/parkjoe/anaconda3/envs/pytorch-ddpm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
  warnings.warn(
/home/parkjoe/anaconda3/envs/pytorch-ddpm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=None`.
  warnings.warn(msg)
Model     : IS: 8.988(0.143), FID:  7.729 
...

I solved the problems by changing sample size and sample step. But your FID (3.249) and IS (9.475) are better than mine. How can I lower the FID from 7.xx to 3.xx through evaluation?

w86763777 commented 3 months ago

Thanks for providing the code. But, I think an error occured during evaluation.

Untitled

I downloaded your checkpoint model and tried to evaluate using this code: CUDA_VISIBLE_DEVICES=0 python main.py --eval --num_images 50000 --flagfile ./logs/DDPM_CIFAR10_EPS/flagfile.txt. And the output script was this:

Files already downloaded and verified
Model params: 34.21 M
generating images: 100%|????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 391/391 [7:53:24<00:00, 72.65s/it]
/home/parkjoe/anaconda3/envs/pytorch-ddpm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.????????????????????????????????????????| 391/391 [7:53:24<00:00, 64.47s/it]
  warnings.warn(
/home/parkjoe/anaconda3/envs/pytorch-ddpm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=None`.
  warnings.warn(msg)
generating images: 100%|????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 391/391 [7:51:47<00:00, 72.40s/it]
0/800000 IS:1.174, IS_std:0.001, FID:460.166, IS_EMA:1.176, IS_std_EMA:0.002, FID_EMA:460.171                                                                                                                                                                                                                       
  7%|???????????????????                                                                                                                                                                                                                                   | 59259/800000 [20:24:53<49:49:44,  4.13it/s, loss=0.029]
7%|???????????????????                                                                                                                                                                                                                                  | 59259/800000 [20:24:53<255:11:17,  1.24s/it, loss=0.029]

At this, I have some questions. First, what do those warning messages mean? Is it normal for those messages to be printed out? Second, why does the FID print out as 460.166? Does this relate to the first question? Third, what is the progress bar that appears after the performance indicators?

The pre-trained model API of torchvision has some changes since version 0.13. The legacy usage is no longer updated, but it does not affect past results.

w86763777 commented 3 months ago

I reuploaded the checkpoint model and retried to evaluate using this code: CUDA_VISIBLE_DEVICES=3 python main.py --flagfile ./logs/DDPM_CIFAR10_EPS/flagfile.txt --notrain --eval. And the output script was this:

generating images: 100%|????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 391/391 [7:52:59<00:00, 72.58s/it]
/home/parkjoe/anaconda3/envs/pytorch-ddpm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
  warnings.warn(
/home/parkjoe/anaconda3/envs/pytorch-ddpm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=None`.
  warnings.warn(msg)
Model     : IS: 8.988(0.143), FID:  7.729 
...

I solved the problems by changing sample size and sample step. But your FID (3.249) and IS (9.475) are better than mine. How can I lower the FID from 7.xx to 3.xx through evaluation?

Can you provide the versions of PyTorch and CUDA you are using? I will try to fix it as soon as possible.

Thanks.

parkjjoe commented 3 months ago

I reuploaded the checkpoint model and retried to evaluate using this code: CUDA_VISIBLE_DEVICES=3 python main.py --flagfile ./logs/DDPM_CIFAR10_EPS/flagfile.txt --notrain --eval. And the output script was this:

generating images: 100%|????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 391/391 [7:52:59<00:00, 72.58s/it]
/home/parkjoe/anaconda3/envs/pytorch-ddpm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
  warnings.warn(
/home/parkjoe/anaconda3/envs/pytorch-ddpm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=None`.
  warnings.warn(msg)
Model     : IS: 8.988(0.143), FID:  7.729 
...

I solved the problems by changing sample size and sample step. But your FID (3.249) and IS (9.475) are better than mine. How can I lower the FID from 7.xx to 3.xx through evaluation?

Can you provide the versions of PyTorch and CUDA you are using? I will try to fix it as soon as possible.

Thanks.

Here are the versions of PyTorch and CUDA.

w86763777 commented 3 months ago

I have tested the pretrained model with the following versions:

The results are as follows:

Model     : IS: 8.882(0.115), FID:  8.060
Model(EMA): IS: 9.493(0.154), FID:  3.210

See the latest commit f804ccbd58a758b07f79a3b9ecdfb1beb67258f6 for details. Thanks.

parkjjoe commented 3 months ago

Thanks for your reply! In DDPM, the FID came out to be 3.17, but your FID came out to be 3.210 when applying EMA. Was EMA applied in DDPM as well? When I searched for DDPM paper, I cannot find such content.

w86763777 commented 3 months ago

3.17 and 3.21 are close enough. It is possible to get the FID of 3.17 if you run the evaluation several times.

You can find the EMA operation here in the official implementation. As far as i know, they did not mention it in the paper, but it is a common trick for image generative models.

parkjjoe commented 3 months ago

My questions has been resolved. Thank you! I will close this issue.