search

AILab-CVC / SEED-Bench

(CVPR2024)A benchmark for evaluating Multimodal LLMs using multiple-choice questions.
Other
315 stars 12 forks source link

In-Context Example Selection Process #14

Open mustafaadogan opened 11 months ago

mustafaadogan commented 11 months ago

Hello! First and foremost, I'd like to congratulate you on this incredible work. I have a question regarding the creation of the dataset for In-Context Captioning and Interleaved Image-Text Analysis dimensions. How were the in-context examples chosen during this process?

Bohao-Lee commented 11 months ago

First, I utilize GPT-4 to transform the appropriate questions and answers related to Dimension 3 (Instance Attribute) and Dimension 5 (Instance Counting) into declarative statements. Next, I divide the attributes into 20 categories using GPT-4 and allow it to classify the corresponding declarative sentences for each attribute. Lastly, I apply filtering within each category to obtain the final set of questions for in-context captioning.