LLaMA 13B works on a single RTX 4080 16GB

[kcchu]

https://github.com/facebookresearch/llama/issues/79#issuecomment-1465779961

System:

• RTX 4080 16GB
• Intel i7 13700
• 32GB RAM
• Ubuntu 22.04.2 LTS

LLaMA 13B

• It uses > 32 GB of host memory when loading and quantizing, be sure you have enough memory or swap
• VRAM usage: about 15GB
• loading time: 5 min (using swap)
• inference time: 30s

LLaMA 7B

• VRAM usage: about 8.6 GB
• loading time: 34s
• inference time: 20s

[chrisbward]

Using the above methods on 3090 Ti 24GB;

LLaMA 13B - 30 seconds loading (with swap - 50GB), 30 seconds inference