search

rhymes-ai / Aria

Codebase for Aria - an Open Multimodal Native MoE
Apache License 2.0
871 stars 72 forks source link

multiple-choice question? #46

Open sailfish009 opened 3 weeks ago

sailfish009 commented 3 weeks ago

Hello, instead of asking the model to describe the image in detail, I want to ask them to answer by selecting 1 option from several choices. Questions that worked well in llava onevision MLLM are not working in Aria. For example, if I ask them to answer the situation in image P by selecting from A, B, C, D, E, if the correct answer is A, they will answer with an additional comment about the situation A. I want to make sure that the additional answer should never be included, so that no words other than A are added.

LiJunnan1992 commented 3 weeks ago

You may want to try adding this sentence to your question: "Answer with the option's letter from the given choices directly."

sailfish009 commented 2 weeks ago

@LiJunnan1992 Hi, I tried adding the question format you mentioned, but it didn't improve the model's answers. The model still answers randomly without choosing from a list of answers.

Coobiw commented 2 weeks ago

Hi, you may refer to https://github.com/rhymes-ai/Aria/tree/main/examples/nextqa. This is a multiple-choice question-answering dataset. Training, inference, instruction format and option extraction are all included. Hope it helpful for you~

sailfish009 commented 2 weeks ago

@Coobiw Thanks for the answer, I took a quick look at the questions in nextqa and it looks like the model is sensitive to the format of the question.