search

open-compass / VLMEvalKit

Open-source evaluation toolkit of large vision-language models (LVLMs), support 160+ VLMs, 50+ benchmarks
https://huggingface.co/spaces/opencompass/open_vlm_leaderboard
Apache License 2.0
1.34k stars 188 forks source link

cannot use Gemini #435

Closed yansuoyuli closed 2 months ago

yansuoyuli commented 2 months ago

I cannot use Gemini

when use: GeminiProVision 2024-09-02 18:57:19,592 - ChatAPI - ERROR - 404 Gemini 1.0 Pro Vision has been deprecated on July 12, 2024. Consider switching to different model, for example gemini-1.5-flash.

when use: GeminiPro1-5 assert model == 'gemini-1.0-pro' AssertionError

yansuoyuli commented 2 months ago

Hi, how can I incorporate gemini-1.5-flash model in this framework? Or can you please update the Gemini model? Thanks!

PhoenixZ810 commented 2 months ago

Hi,

The latest version of vlmevalkit has has resolved this issue. Please refer to gemini and use Gemini-1.5-pro.

Thank you!

PhoenixZ810 commented 2 months ago

Hi, how can I incorporate gemini-1.5-flash model in this framework? Or can you please update the Gemini model? Thanks!

We will consider adding support for the gemini-1.5-flash model soon.