search

open-compass / VLMEvalKit

Open-source evaluation toolkit of large vision-language models (LVLMs), support ~100 VLMs, 40+ benchmarks
https://huggingface.co/spaces/opencompass/open_vlm_leaderboard
Apache License 2.0
1.25k stars 178 forks source link

InternVL2 Error #289

Closed amitbcp closed 2 months ago

amitbcp commented 3 months ago 
Generating with V2.0
A: torch.Size([7175, 3200]), B: torch.Size([9600, 3200]), C: (7175, 9600); (lda, ldb, ldc): (c_int(229600), c_int(307200), c_int(229600)); (m, n, k): (c_int(7175), c_int(9600), c_int(3200))
cuBLAS API failed with status 15
error detected

Command - 

python3  run.py --data MMBench_DEV_EN MME SEEDBench_IMG MMMU_DEV_VAL --model InternVL2-40B  --verbose

I am able to run InternVL series 1 but for InternVL2 I am getting this error.Though I am able to run the 26B InternVL2

I have build the VLMEvalKit from its latest master branch.

Also the config does not have the InternVL2-76B listed which has been integrated in the code

junming-yang commented 3 months ago

InternVL2 uses transformers == 4.37.0. Please check your package version.

InternVL2-76B is under testing. We will officially support and update it later in the config.py file.

amitbcp commented 3 months ago

@junming-yang , yes I have the correct transformers version. InternVL2-26B runs but InternVL2-4)b gives the type mismatch error

junming-yang commented 3 months ago

I tried to reproduce this bug using the latest code and ran the same command. It appears to be working normally.

kennymckormick commented 3 months ago

Hi, @amitbcp , InternVL2-76B is already supported now. For the type mismatch error, please provide more context.

amitbcp commented 3 months ago

Thanks @kennymckormick ! yup was able to run InternVL2 thought the leaderboard is not updated with the same.

I have the benchmark results for InternVL2-76B let me know how can I share ?