bitsandbytes-foundation / bitsandbytes

Accessible large language models via k-bit quantization for PyTorch.
https://huggingface.co/docs/bitsandbytes/main/en/index
MIT License
6.31k stars 634 forks source link

Nf4 reload #1348

Closed jiqing-feng closed 2 months ago

jiqing-feng commented 2 months ago

Fix nf4 memory issue when using state_dict().

Still has memory issue when loading nf4 model.

jiqing-feng commented 2 months ago