enable cpu bnb distributed lora finetune

huggingface / accelerate

🚀 A simple way to launch, train, and use PyTorch models on almost any device and distributed configuration, automatic mixed precision (including fp8), and easy-to-configure FSDP and DeepSpeed support

https://huggingface.co/docs/accelerate

Apache License 2.0

7.98k stars 975 forks source link

enable cpu bnb distributed lora finetune #3159

Closed jiqing-feng closed 1 month ago

jiqing-feng commented 1 month ago

Hi @SunMarc , this PR enables cpu bnb model distributed finetune by removing check index because we don't need index in cpu device. Please review it. Thanks!

jiqing-feng commented 1 month ago

Hi @SunMarc @ArthurZucker , if we don't make this change, the error be like:

HuggingFaceDocBuilderDev commented 1 month ago

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

jiqing-feng commented 1 month ago

Hi @SunMarc . I have checked bnb multi-backend, please review it. Thx!