the nuimage 256x256 recurrence

[Sirius-Li]

The pretrained nuimage 256x256 model pretrained model generates 14772x5 = 15485 images for testing, and then calculates the fid with the val set 14772 images, while the fid is 19.48. Moreover, I retrain the nuimage 256x256 model on stable diffusion which reach 15.90, also not reaching the accuracy of paper(14.58). I resize the images to 256x256 and use fid_score of pytorch_fid package to calculate the fid. Is this correct ?

[KaiChen1998]

As claimed in our README file, we recommend users follow the exact setting of the LAMA repo step by step for precise evaluation on the COCO-Stuff Layout-to-Image benchmark, since the FID calculation is extremely sensitive to the implementation details.

Currently, I notice the following differences:

• Following LAMA, we filter the original COCO validation set, resulting in a validation set of 3,097 images. Therefore, we generate 3097x5=15485, and then calculate FID with the real 3097 images. We have provided the filtered validation image list in HuggingFace.
• Check more details in the Section 5.1 of the LAMA paper.
• LAMA uses a tensorflow-based FID calculation code instead of pytorch_fid.
• Frankly speaking, there is not much difference between 15.90 FID and 14.58 FID.