search

keras-team / keras-hub

Pretrained model hub for Keras 3
Apache License 2.0
790 stars 242 forks source link

Any plans for QLora? #1537

Closed asmith26 closed 3 weeks ago

asmith26 commented 7 months ago

Hi there! keras-nlp supports Lora, e.g. from https://ai.google.dev/gemma/docs/lora_tuning

...
gemma_lm.backbone.enable_lora(rank=4)
...

Just wondering are there any plans to implement QLora finetuning, to help make finetuning even more accessible on smaller GPUs? https://arxiv.org/abs/2305.14314 https://huggingface.co/blog/4bit-transformers-bitsandbytes

Many thanks for any help, and this lib!

mattdangerw commented 7 months ago

Yes actually! We are currently working or rolling out some support for quantization generally and quantize lora.

We will work on some actual docs for this soon, but for now you can check out the qlora.py in the description of this PR to see what we are working on -> https://github.com/keras-team/keras/pull/19356/

I believe the 4 bit normal float from the qlora paper is not currently available, but int8 quantized weights layered in with lora should be good to go.

Early days on the feature, so any feedback appreciated as we build it out.

github-actions[bot] commented 6 months ago

This issue is stale because it has been open for 14 days with no activity. It will be closed if no further activity occurs. Thank you.

mattdangerw commented 3 months ago

Guide on this incoming...

https://github.com/keras-team/keras-io/pull/1889

Gopi-Uppari commented 1 month ago

Hi @asmith26,

Could you please confirm if this issue is resolved for you with the above comment ? Please feel free to close the issue if it is resolved ?

Thank you.

asmith26 commented 3 weeks ago

Have got this working https://keras.io/examples/keras_recipes/parameter_efficient_finetuning_of_gemma_with_lora_and_qlora/#qlora-finetuning

Many thanks again!