Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs

[runhani]

Paper : https://arxiv.org/pdf/2406.16860 Website : https://cambrian-mllm.github.io Code : https://github.com/cambrian-mllm/cambrian Models : https://huggingface.co/nyu-visionx/ Data : https://huggingface.co/datasets/nyu-visionx/Cambrian-10M CV-Bench : https://huggingface.co/datasets/nyu-visionx/CV-Bench Evaluation : https://github.com/cambrian-mllm/cambrian

Open cookbook for instruction-tuned MLLMs

Intro

• 인간의 지식은 어디로 부터 오는가?
• sight, hearing, touch, taste, smell, interactions with the physical world.

Data

• Instruction Tuning Data :

Becnmark

• CV-Bench = vision-centric MLLM benchmark

LLMs

• LLaMA-3-Instruct-8B
• Vicuna-1.5-13B
• Hermes-2-Yi-34B

Vision

• OpenAI CLIP ViT-L/14@336
• SigLIP ViT-SO400M/14@384
• OpenCLIP ConvNeXt-XXL@1024
• DINOv2 ViT-L/14@518

Connector

• Spatial Vision Aggregator

[runhani]

MLLM의 중요한 5가지 key components

LLM

Visual Encoder

Multimodal Connector

• Resamplers, Q-Formers, MLP Projectors 크게 3가지가 있다.
• 문제는 이미지 해상도가 커질수록 사용하는 token이 너무 많다는 문제가 있다.
• LLaVA-Next 672 pixels → 2880 tokens
• 그래서 high-resolution 이미지여도 small 개수의 tokens로 mapping하는 connector 제안

Data Curation Pipeline

Instruction Tuning Strategy

[runhani]

누가 답변하는거야? LLM vs MLLM

[runhani]

becnmark를 clustering해 봅시다.

• “General” in green
• “Knowledge” in yellow
• “Chart & OCR” in red
• “Vision-Centric” in blue