search

meta-llama / llama-models

Utilities intended for use with Llama models.
Other
4.92k stars 843 forks source link

[Request] Preview model disk size usage #88

Open czlr opened 3 months ago

czlr commented 3 months ago

Request: As a casual user without much knowledge in LLMs, it would be nice to know upfront how much disk space the models need.

Currently: The various posts and docs only mention that llama 3.1 comes in different variants: 8B, 70B, 405B; regular vs instruct; etc. But they don't mention things like required disk size usage, or minimum system specs to run things smoothly. (Unless I accidentally missed those details) I can only start a download blindly and monitor while it's going. The 70B model also seems to come in 17GB .pth parts -- I can't see how many parts are remaining either.

DAOZHENREN commented 3 months ago

For the 70B model, it may take 131GB

DAOZHENREN commented 3 months ago

And 15GB for the 8B model. The regular and instruct are equal in size, but the latter is finetuned toward chat and is safer(less toxic).

vonpetersenn commented 2 weeks ago

I have a related question: if the model has 70B parameters, wouldn't we expect the size of the model to be 70*4=280 GB? A Float32 parameter takes up 4 bytes of storage. Are the parameters of LLaMA Float16-numbers?

DAOZHENREN commented 2 weeks ago

I think that you have misunderstood the 'B' in the 70B. Actually, it means billion.

ashwinb commented 2 weeks ago

@vonpetersenn Yes they are float16 -- specifically "bf16" (Brain Float 16) format.

vonpetersenn commented 2 weeks ago

thank you very much!