When will Hugging Face support QDORA?

NVlabs / DoRA

[ICML2024 (Oral)] Official PyTorch implementation of DoRA: Weight-Decomposed Low-Rank Adaptation

https://nbasyl.github.io/DoRA-project-page/

Other

588 stars 36 forks source link

When will Hugging Face support QDORA? #8

Closed 190679163 closed 4 months ago

190679163 commented 4 months ago

I'm thrilled that DORA is available on Hugging Face PEFT. When will QDORA be supported as well?

nbasyl commented 4 months ago

Hi, thanks for your interest in our work, currently, the Hugging Face PEFT's support for QDoRA still has some bugs, so I recommend using the FSDP/QLoRA for QDoRA which you can refer to my step by step instructions under the QDoRA folder