How to merge Qlora FSDP weights with an LLM and save model.

huggingface / accelerate

🚀 A simple way to launch, train, and use PyTorch models on almost any device and distributed configuration, automatic mixed precision (including fp8), and easy-to-configure FSDP and DeepSpeed support

https://huggingface.co/docs/accelerate

Apache License 2.0

7.32k stars 872 forks source link

How to merge Qlora FSDP weights with an LLM and save model. #2904

Closed Minami-su closed 1 day ago

BenjaminBossan commented 1 day ago

I assume that you currently get an error when you try? What is your use case that requires to merge the QLoRA adapter before saving the model?

Minami-su commented 1 day ago

https://github.com/AnswerDotAI/fsdp_qlora/blob/main/Converting%20the%20State%20Dict.ipynb