search

crowsonkb / k-diffusion

Karras et al. (2022) diffusion models for PyTorch
MIT License
2.31k stars 375 forks source link

FID increases with training #3

Closed jwliu-cc closed 2 years ago

jwliu-cc commented 2 years ago

I just run train_cifar.py on a single GPU and found that the FID increased from 29.68 (50k steps) to 45.16 (260k steps). Is there any idea for this?

crowsonkb commented 2 years ago

Yes, train_cifar.py needs to be folded into the main training script train.py which uses non-leaky augmentations and dropout to prevent overfitting. If you modify train.py to use the CIFAR-10 dataset and try that it should overfit less or not at all (I have done tests on small datasets and these modifications have prevented overfitting).

crowsonkb commented 2 years ago

Figure_1

Here is FID vs step for 128x128 ArtBench-10 (60k images). The blue line is Karras Config E without dropout, the orange line is Karras Config F (non-leaky augmentation probability 15%) with dropout 0.05.

jwliu-cc commented 2 years ago

Thank you very much for your reply, i have tried train.py with dropout on cifar10, the FID reduced to 27.74 after 130k steps. model_demo_00130000