search

artidoro / qlora

QLoRA: Efficient Finetuning of Quantized LLMs
https://arxiv.org/abs/2305.14314
MIT License
9.96k stars 820 forks source link

Applicable to StarCoder? #121

Open SinanAkkoyun opened 1 year ago

SinanAkkoyun commented 1 year ago

Hello, thank you for the awesome work!!!

What would I need to do/modify to make this work with StarCoder Base?

Thank you!

dmckinno commented 1 year ago

Does load_in_4bit need to be implemented for each model type? I get this error: TypeError: GPTBigCodeForCausalLM.__init__() got an unexpected keyword argument 'load_in_4bit'

dmckinno commented 1 year ago

I got this working. The issue is that the 4-bit integration hasn't been pulled into the accelerate or transformers releases on pypy yet.

If you upgrade both to main (accelerate-0.20.0.dev0 and transformers-4.30.0.dev0), you will be good to go.