Reproducing all numbers in Tab 3

THUDM / ImageReward

[NeurIPS 2023] ImageReward: Learning and Evaluating Human Preferences for Text-to-image Generation

Apache License 2.0

1.18k stars 65 forks source link

Reproducing all numbers in Tab 3 #78

Open vishaal27 opened 7 months ago

vishaal27 commented 7 months ago

Hey, thanks for your great work.

I am able to reproduce the CLIP-Score and ImageReward numbers for the Preference Acc. column of tab. 3 of your paper using the test.py script. However, there is no script to run the evaluations for the other columns i.e. Recall and Filter. Could you please advise on how to reproduce these numbers as well? I think the data for reproducing that is not released yet right?

Thanks!

xujz18 commented 7 months ago

The test set seems to be lost when we changed our server. I will release it if we find the test set. If necessary, you may try sampling from test set of ImageRewardDB yourself.