Consult the 'circle eval' code

open-compass / MMBench

Official Repo of "MMBench: Is Your Multi-modal Model an All-around Player?"

Apache License 2.0

164 stars 10 forks source link

Consult the 'circle eval' code #29

Closed Baiqi-Li closed 5 months ago

Baiqi-Li commented 7 months ago

Your work is great! Where can I find the code related to circle eval?

kennymckormick commented 6 months ago

Hi, @Baiqi-Li , CircularEval is implemented in VLMEvalKit, check this file: https://github.com/open-compass/VLMEvalKit/blob/main/vlmeval/evaluate/multiple_choice.py BTW, note that our circular eval implementations requires the data file to include all the circular passes (in the data file of MMBench, each circular pass is indexed with question_index + circular_index * 1e6). You can download and parse the data files to see more details.