EvolvingLMMs-Lab lmms-eval issues

EvolvingLMMs-Lab / lmms-eval

Accelerating the development of large multimodal models (LMMs) with lmms-eval

https://lmms-lab.github.io/

Other

991 stars 47 forks source link

issues

Newest

Newest Most commented Recently updated Oldest Least commented Least recently updated

Can not load llava_hf models since the new updates!

#119 hasanar1f opened 12 hours ago
0
Fix the potential risk by PR #117

#118 teowu closed 2 days ago
0
LongVideoBench for LMMs-Eval

#117 teowu closed 3 days ago
0
There is an inconsistency in the number of mmbench images

#116 MuskAI opened 3 days ago
1
Unable to reproduce SQA results for llava-1.5

#115 clairez-cerebras opened 3 days ago
1
add tinyllava

#114 zjysteven closed 7 hours ago
1
Q-Bench, Q-Bench2, A-Bench

#113 teowu closed 3 days ago
0
Error during videomme

#112 yukang2017 opened 5 days ago
1
add II-Bench

#111 XinrunDu closed 4 days ago
0
building context takes too long

#110 simplelifetime opened 5 days ago
1
[Small Update] Update the version of LMMs-Eval

#109 pufanyi closed 5 days ago
0
[Upgrade to v0.2] Embracing Video Evaluations with LMMs-Eval

#108 Luodian closed 5 days ago
0
update gpt-3.5-turbo version

#107 AtsuMiyai closed 6 days ago
0
AssertionError: No tasks specified, or no tasks found. Please verify the task names when using local dataset path

#106 MuskAI opened 1 week ago
0
Include VCR

#105 tianyu-z closed 5 days ago
4
Why cot not supported for Mathverse

#104 huiyeruzhou opened 1 week ago
0
Unreasonable data in AI2D dataset used for evaluation

#103 yqy2001 opened 1 week ago
3
Allow loading models and tasks from external packages

#102 lorenzomammana opened 1 week ago
1
Update conbench in README

#101 Gumpest closed 1 week ago
0
add Conbench

#100 Gumpest closed 1 week ago
1
How to contribute a new dataset?

#99 JohnTang93 opened 1 week ago
1
qwenvl-7b evaluate refcoco|+|g cider and IOU are all None,

#98 AderonHuang opened 2 weeks ago
10
Add MathVerse in README.md

#97 CaraJ7 closed 2 weeks ago
0
Any plan for supporting Lora peft loading

#96 hxhcreate opened 2 weeks ago
1
add MM-UPD

#95 AtsuMiyai closed 2 weeks ago
6
Error when import InstructBLIP

#94 hanqiu-hq opened 3 weeks ago
1
Add m3exam

#93 Jiawei-Guo opened 3 weeks ago
0
add multi-lingual MMMU tasks

#92 Junpliu closed 3 weeks ago
0
how to use “few shot”

#91 woshidengweimo opened 3 weeks ago
1
Prompt for MathVista

#90 yqy2001 closed 2 weeks ago
2
W&B Logging Issue on MMMU & Wrong parsed_pred

#89 AlekseyKorshuk opened 3 weeks ago
1
No "llava_llama_3" template

#88 yqy2001 opened 3 weeks ago
2
Adding microsoft/Phi-3-vision-128k-instruct model.

#87 vfragoso closed 3 weeks ago
0
Question on evaluation across multiple tasks

#86 AtsuMiyai closed 3 weeks ago
0
how to get three pope results [rad,pop,adv] with lmms-eval?

#85 baiyuting closed 3 weeks ago
2
[Week 1] Adding 5 examples for Hindi

#84 simran-khanuja closed 3 weeks ago
0
[Feature Request] Evaluating quantized models

#83 zjysteven closed 3 weeks ago
3
Error while testing Qwen on POPE

#82 hxhcreate closed 2 weeks ago
1
LLaVA Benchmark

#81 yqy2001 opened 1 month ago
2
Extremely Low Performance with LLaVA-1.5-7B on MM-Vet Benchmark

#80 lambert-x opened 1 month ago
6
[MultiGPU Evaluation Error]: model on differencet device

#79 hxhcreate closed 1 month ago
2
"llava_hf" is not registered

#78 kyleliang919 opened 1 month ago
4
Too many requests for MathVista

#77 baichuanzhou opened 1 month ago
0
Update llava.py - Llama 3 Conversation Messages attribute copied as an immutable tuple

#76 jkitaok opened 1 month ago
1
An issue during evaluating MathVista

#75 simplelifetime opened 1 month ago
1
Support for llava-hf/llava-v1.6-x

#74 justinphan3110cais opened 1 month ago
3
[Feat] Add qwen vl api

#73 kcz358 closed 1 month ago
1
Multinode Evaluation

#72 yqy2001 opened 1 month ago
2
While this lmms-eval support V*Bench?

#71 xmu-xiaoma666 opened 1 month ago
0
Bugfix: WebSRC should be token-level F1 NOT character-level

#70 hunterheiden closed 1 month ago
0