search

cambrian-mllm / cambrian

Cambrian-1 is a family of multimodal LLMs with a vision-centric design.
https://cambrian-mllm.github.io/
Apache License 2.0
1.77k stars 115 forks source link

CV-Bench Evaluation #81

Closed nku-zhichengzhang closed 2 weeks ago

nku-zhichengzhang commented 1 month ago

Congrats to the Cambrian team on your acceptance to NeurIPS! This is a significant achievement, and it's a testament to the hard work and innovation you've put into the project.

Issue Description

While exploring the repository, I noticed that there seems to be no inference and evaluation codes for cv-bench in the eval/eval folder.

Questions

  1. Is the evaluation code for cv-bench located elsewhere in the repository?
  2. If not, are there plans to add this code to the eval/eval folder or another appropriate location?
  3. Is there any specific reason for not including the cv-bench evaluation code at this time?
Jike338 commented 4 weeks ago

same ask here :))

WayneJin0918 commented 3 weeks ago

CVbench2D is the average of COCO and ADE in eval. CVbench3D is the Omni in eval. You can find it directly.

WayneJin0918 commented 3 weeks ago

https://github.com/cambrian-mllm/cambrian/tree/main/eval/eval/ade https://github.com/cambrian-mllm/cambrian/tree/main/eval/eval/coco https://github.com/cambrian-mllm/cambrian/tree/main/eval/eval/omni for details.

nku-zhichengzhang commented 3 weeks ago

Thx for details~