How much RAM needed to convert gpt2 13b model to ggml using your Manual convert function?

LLukas22 / llm-rs-python

Unofficial python bindings for the rust llm library. 🐍❤️🦀

MIT License

71 stars 4 forks source link

How much RAM needed to convert gpt2 13b model to ggml using your Manual convert function? #33

Open JohnClaw opened 11 months ago

JohnClaw commented 11 months ago

I'm trying to convert it on 16gb RAM but converting process seems to last forever.

LLukas22 commented 11 months ago

Well you can calculate it via: 13b times 16 Bit (f16) = 26 GB. Accelerate will probably try to page some of the layers, if you exceed your 16 GB and get stuck there. Theoretically it's possible to stream the layers in, but i think neither GGML or this project has implemented that yet for GPT 2.