search

Rayhane-mamah / Efficient-VDVAE

Official Pytorch and JAX implementation of "Efficient-VDVAE: Less is more"
https://arxiv.org/abs/2203.13751
MIT License
190 stars 23 forks source link

Any plan to compute FID&IS scores? #2

Open Kim-Dongjun opened 2 years ago

Kim-Dongjun commented 2 years ago

Hi! It is a great pleasure to meet an incredible work! I appreciate your dedication to this community.

I have checked the paper and github, but it seems that there is no sample generation performance, such as Frechet Inception Distance (FID) or Inception Score (IS). Is there any plan to compute FID or IS in near future?

Thank you, Dongjun Kim.

Rayhane-mamah commented 2 years ago

Hi @Kim-Dongjun!

Thank you for your kind words! We also hold great respect for your work on UDM.

The main goal of our work is to make Very deep VAEs, which are log-likelihood models in essence, more efficient and accessible in general. We did consider adding sample quality performance in our metrics (such as FID or IS) as means to compare the VDVAE performance with other generative models, but we had one concern: FID (and IS) is a biased metric by design and is sensitive to numerous subtle choices in their low-level image processing methods (reference).

Since claiming better generative (or log-likelihood) performance is not the primary goal of this work, we decided to not report FID/IS in the paper and instead focus on them in possible future work that tries to improve VAEs generative quality.

Generally speaking however, one should expect Efficient-VDVAE's FID to be similar to VDVAE's FID as reported in this work. As you well know, it is common for VAEs to have worse sample quality than GANs or Diffusion models (on average), and some recent work is trying to describe these problems and solve them (dpVAE, NCP-VAE).

With that said, if there is a strong desire to measure the sample generation performance of our models, we can add the utility scripts to do so, and add the FID/IS metrics in the README. However, our priority still remains to provide pre-trained model checkpoints first.

Thank you, Rayhane Mama.

Kim-Dongjun commented 2 years ago

Dear @Rayhane-mamah ,

Thank you for your fast and kind reply :) I completely understand your priority on tasks! I hope your work to become a VAE's cornerstone to break GAN's generation performance sooner or later. It was an honor to meet your work.

Thank you, Dongjun Kim.

Vanlogh commented 2 years ago

Dear @Kim-Dongjun,

Thank you for your very nice words. We're very happy of your feedback on FID scores. We're adding this to our potential TO-DOs as we deem this important for future work. The issue of the generative power of VAEs is likely going to keep this community busy for a little while. Looking forward to your next exciting research!

Thank you, Louay Hazami

MultiPath commented 2 years ago

I think adding FID or any quality metric is necessary, even though they are somewhat biased. It is a good way to know where VAE-based methods are set in the generative model families.

Vanlogh commented 2 years ago

Hello @MultiPath,

Thank you for your comment. I do agree that adding the FID would be helpful. Unfortunately, however, we have moved on from this project, so we simply don't have the bandwidth to do this at the moment.

P.S: A recent work which I found quite interesting has dove into this and the results seem to improve the FID scores of SotA VAEs https://arxiv.org/abs/2210.10205