search

EvolvingLMMs-Lab / lmms-eval

Accelerating the development of large multimodal models (LMMs) with lmms-eval
https://lmms-lab.github.io/
Other
1.04k stars 55 forks source link

latency and throughput (tokens/sec) in evaluation #131

Open hasanar1f opened 5 days ago

hasanar1f commented 5 days ago

Do you have a plan to include metrics like latency of generation and throughput (tokens/sec) in the evaluation? I think this would be a good addition. Having these system evaluations will surely help the development and research on efficiency vs. performance

zjysteven commented 4 days ago

Was wondering the same!

Luodian commented 4 days ago

Oh, that's a good suggestion, thank you for proposing that haha.

We will list this feature in our dev plan.

hasanar1f commented 4 days ago

thanks!!