search

EvolvingLMMs-Lab / lmms-eval

Accelerating the development of large multimodal models (LMMs) with lmms-eval
https://lmms-lab.github.io/
Other
1.02k stars 52 forks source link

Why cot not supported for Mathverse #104

Open huiyeruzhou opened 2 weeks ago

huiyeruzhou commented 2 weeks ago

Hi! I'm looking the evaluation config of mathverse in lmms-eval/lmms_eval/tasks/mathverse/mathverse_testmini.yaml 

model_specific_prompt_kwargs:
  default:
    shot_type: "format-prompt" # can also be "custom-prompt"
    query_type: "query_wo" # now only support query_wo

What will happen if query_type is specified as ”query_cot"? When I read MathVerseEvaluator.create_one_query, it seems that change query_type will just change the input prompt.

        # question
        if shot_type == "format-prompt":
            question_text = f"{problem[query_type]}"

I guess that maybe the difficulty is to implement the evaluation strategy for cot? Thanks for any helpful comment.