search

THUDM / ImageReward

[NeurIPS 2023] ImageReward: Learning and Evaluating Human Preferences for Text-to-image Generation
Apache License 2.0
1.18k stars 65 forks source link

[dataset] Collecting rankings from different annotators #85

Open dain5832 opened 5 months ago

dain5832 commented 5 months ago

Hello, I wonder how you collected the rankings from different annotators, especially when there's a tie. For instance, say there are six annotators, and three ranked an image as rank2 and another three ranked as rank3. Then, what would be the final ranking of the image, 2 or 3??

xujz18 commented 3 months ago

Hello, thanks for your attention! In this case, we don't use a pair of tie for training.