search

open-compass / VLMEvalKit

Open-source evaluation toolkit of large vision-language models (LVLMs), support 160+ VLMs, 50+ benchmarks
https://huggingface.co/spaces/opencompass/open_vlm_leaderboard
Apache License 2.0
1.34k stars 188 forks source link

Custom prompt for API? #444

Closed geweihgg closed 2 months ago

geweihgg commented 2 months ago

I find that testing models via API is not consistent with testing models directly, especially the prompt. API do not support custom prompt now. Will VLMEvalKit support custom prompt for API?

testing API：https://github.com/open-compass/VLMEvalKit/blob/main/vlmeval/inference.py#L33 testing model：https://github.com/open-compass/VLMEvalKit/blob/main/vlmeval/inference.py#L111

kennymckormick commented 2 months ago

Hi, @geweihgg , We have added custom prompt for APIs now. Please have another try.