search

keras-team / keras-hub

Pretrained model hub for Keras 3
Apache License 2.0
803 stars 243 forks source link

Does Keras-NLP support (Q)Dora? #1794

Open asmith26 opened 3 months ago

asmith26 commented 3 months ago

Hi there,

I recently came across this blog post: https://www.answer.ai/posts/2024-04-26-fsdp-qdora-llama3.html

[QDoRA] is just as memory efficient and scalable as FSDP/QLoRA, and critically is also as accurate for continued pre-training as full weight training.

Does Keras-NLP support/plan to support (Q)Dora?

Many thanks for any help, and this lib!