CLEVR pretrained model gives FID 22

[JanRocketMan]

Hi, kudos for great work!

I've just noticed that with recommended preprocessing and evaluation, the metrics on gdrive:cityscapes work as expected (FID ~5.2), while for CLEVR exactly two same lines:

python prepare_data.py --clevr --max-images 100000
python run_network.py --eval --gpus 0 --expname clevr-exp --dataset clevr --pretrained-pkl gdrive:clevr-snapshot.pkl

give ~22 FID, not 9.2. Can you please double-check if the provided snapshot is correct? Or am I missing smth here?

Thanks in advance!

[dorarad]

Hi, thank you very much for pointing that out! I'll check that today. I verified all scores locally so there must be a bug / incorrect snapshot file. Will get back shortly!

[dorarad]

Alright I checked. The source of the issue was a change in the way the dataset images are preprocessed (this line in particular https://github.com/dorarad/gansformer/blob/main/training/misc.py#L137) that caused the images the model is evaluated on to look different than the images it was trained on (using different settings for image ratio and cropping than the original settings used). It should be fixed now, so I recommend rerunning python prepare_data.py --clevr --max-images 100000 to produce the new fixed data and then running the evaluation again on that.

I'm verifying it now too so will be able to follow up in couple hours with a confirmation of whether it resolves the problem.

[dorarad]

Alright, I verified that numbers are now well. Let me know if you still experience the issue.

[JanRocketMan]

Hmm strange, after this my FID reduced to 12.8, but there's still a gap and I can't figure out why...

[dorarad]

Will look further into it!

[JanRocketMan]

Could you send please your md5 hashes of the .tfrecords files as well? So that we could verify whether it's a data issue. Btw samples look just fine

[dorarad]

Yea the issue was just with the data preprocessing (the cropping settings). I'll make the original tfrecords available for download (as is the case currently with FFHQ and Cityscapes) so to ensure perfect match.

[dorarad]

Alright I uploaded the data (the difference between the locally processed data that leads to the FID score 12 is that the images ratio height/weight in the data is 2/3 while the pretrained model was trained on images with ratio 3/4. You can try again to run python prepare_data.py --clevr --max-images 100000 should take a few minutes only (because will just download the data) and then another 20min to run the python run_network.py --eval --gpus 0 --expname clevr-exp --dataset clevr --pretrained-pkl gdrive:clevr-snapshot.pkl . Let me know if you're still getting higher FID!