search

cambrian-mllm / cambrian

Cambrian-1 is a family of multimodal LLMs with a vision-centric design.
https://cambrian-mllm.github.io/
Apache License 2.0
1.4k stars 88 forks source link

why cam34b weight sum 130G? #19

Closed CSEEduanyu closed 2 days ago

CSEEduanyu commented 3 days ago

https://huggingface.co/nyu-visionx/cambrian-34b/tree/main

CSEEduanyu commented 3 days ago

The actual load of the LLM part may be 20B?

ellisbrown commented 2 days ago

the LLM base for this model is Yi-34B, which totals ~70GB using the bfloat16 dtype https://huggingface.co/NousResearch/Nous-Hermes-2-Yi-34B/blob/fcb0a8847e76aea14aba9aa44009d4418ad7c18f/config.json#L25

our model is stored in float32, which is likely the cause of the increased size https://huggingface.co/nyu-visionx/cambrian-34b/blob/7fc61cdc8e5207071c5c3a842c9cb181aa1cfbc7/config.json#L64

CSEEduanyu commented 1 day ago

the LLM base for this model is Yi-34B, which totals ~70GB using the bfloat16 dtype https://huggingface.co/NousResearch/Nous-Hermes-2-Yi-34B/blob/fcb0a8847e76aea14aba9aa44009d4418ad7c18f/config.json#L25

our model is stored in float32, which is likely the cause of the increased size https://huggingface.co/nyu-visionx/cambrian-34b/blob/7fc61cdc8e5207071c5c3a842c9cb181aa1cfbc7/config.json#L64

I didn't change config. Why did I successfully load the model in A100*1? Only 80 gigabytes of video memory.

CSEEduanyu commented 1 day ago

the LLM base for this model is Yi-34B, which totals ~70GB using the bfloat16 dtype https://huggingface.co/NousResearch/Nous-Hermes-2-Yi-34B/blob/fcb0a8847e76aea14aba9aa44009d4418ad7c18f/config.json#L25

our model is stored in float32, which is likely the cause of the increased size https://huggingface.co/nyu-visionx/cambrian-34b/blob/7fc61cdc8e5207071c5c3a842c9cb181aa1cfbc7/config.json#L64

The other question is, if you change the dtype to fp16, what will happen?