ShiFengyuan1999
commented
2 weeks ago Thanks for your interest in our work.
Yes, we use different training data. Their models are trained on Open-Images and we only use ImageNet for training (we are planning to scale-up training data). However, when testing both on ImageNet, we achieved much better reconstruction results.
Specifically,
- They provide VQVAE checkpoints at a down sampling rates of 4/8/16, and we provide 8/16. They receive the optimal results of 0.58 rFID at 4x while we obtain 0.39 rFID at 8x. Note that smaller downsample ratios make more tokens, and lead to better reconstruction. So we may achieve even better results if we also use 4x downsampling.
- They also provide non-VQ version for diffusion model training. However, only the 4x downsampling version achieve better results than our VQ model, which means that we achieved better results even with visual quantization at the same downsampling rate 😆. We do not plan to train non-VQ models (sole VAE) since our main focus is autoregressive visual generation which needs visual tokens for training.
When choosing tokenizers for your own project, we recommend to take both the reconstruction capability and downsampling rate into consideration. Smaller downsampling rate generally lead to better reconstruction, however, the large number of tokens may make Transformer hard to learn.
Hope to provide some insights for you.
yxgeee
StarCycle
Hi,
Do you compare your ckpt with the vae/vqvae here?
If the direct comparison is not reasonable because their ckpts are trained with more data, did they have a version that was only trained with imagenet data but with the same config? Or can you evaluate their ckpt on imagenet even if these are trained on more data...I just want to know which ckpt is most suitable for my current application.
Best, StarCycle