125G of memory, executing merge-weights.py on 30B will oom

randaller / llama-chat

Chat with Meta's LLaMA models at home made easy

GNU General Public License v3.0

833 stars 118 forks source link

125G of memory, executing merge-weights.py on 30B will oom #5

Open Chting opened 1 year ago

ortegaalfredo commented 1 year ago

Set up at least a 40G swap, it need about 130G of memory for merging 30B

Chting commented 1 year ago

Yes, I set 100G SWP, I don't know if 65B is enough

CaullenOmdahl commented 1 year ago

I was successful. My system has 3 separate swap files. 1 ZRAM (ram compression), 1 on an NVME SSD (32GB), and 1 on a PCI-e SSD (4GB). Set the Zram and NVME priority 1 and PCI-e Priority -1. Ubuntu 22.04, Ryzen 5900x, 64GB RAM.

randaller commented 1 year ago

Mine swap is set to 256 Gb in addition to a 128 Gb of RAM. Max swap usage was about 180 Gb while merging 65B model.