search

layer6ai-labs / dgm-eval

Codebase for evaluation of deep generative models as presented in Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models
MIT License
123 stars 9 forks source link

Question about dataset resize #3

Closed yjhong89 closed 1 year ago

yjhong89 commented 1 year ago

Hello, I am YJ.Hong and thanks for sharing great work!

There is a clean_resize option in dataset transformation https://github.com/layer6ai-labs/dgm-eval/blob/d57a6d2d6bd613332dd21696994afff8efa78575/dgm_eval/models/dinov2.py#L51 and how it differs from TF.Resize(224,224) ??

georgestein commented 1 year ago

Hi there, thanks for the question!

The clean_resize option was added to our codebase to match https://github.com/GaParmar/clean-fid, which showed that the PyTorch image resizing/interpolation does not match the resizing in PIL, which is particularly problematic when downsampling images to very small sizes.

We experimented with both options and found little to no difference in practice when using DINOv2 as the encoder. Given that DINOv2 was trained with PyTorch image resizing and not PIL, we chose to use the same resizing that DINOv2 was trained on, and did not use the clean_resize option to measure the results in our leaderboard tables. As such, clean_resize defaults to False, and the code uses PyTorch resizing.

yjhong89 commented 12 months ago

Thanks for answer!