search

yxli2123 / LoftQ

MIT License
193 stars 18 forks source link

Is there any way for using LoftQ to GPTQ or AWQ model? #18

Open LameloBally opened 7 months ago

LameloBally commented 7 months ago

I want to use LoftQ initialization for GPTQ or AWQ baseline model.

Is that possible?

yxli2123 commented 6 months ago

Hi @LameloBally, thanks for your interest of our work.

Unfortunately the answer is no. LoftQ aims to provide a data-free initialization for all downstream task finetuning, so quantization with data calibration is out of our scope. However, we welcome you to explore more possibilities of our method: you can obtain a quantized backbone (Q) from GPTQ or AWQ and then initialize LoRA adapters, A and B, by SVD(W-Q), where W is the full-precision pre-trained weight.

Let me know if you have further questions.

LameloBally commented 6 months ago

@yxli2123 Thanks for your kind answer. It was very helpful for me!! But, I have another question. How Can I initialize LoRA Adapter by SVD(W-Q) s.t. Q is GPTQ model & W is full precision. I thought LoftQ is very innovative idea so I want to apply it to other cases to develop it further. It will be very helpful for me if you let me know How can I find any guide or the way I can do it.

Thanks!