search

project-baize / baize-chatbot

Let ChatGPT teach your own chatbot in hours with a single GPU!
https://arxiv.org/abs/2304.01196
GNU General Public License v3.0
3.17k stars 287 forks source link

Features : Can it use 4 bit QLORA ? #46

Open x4080 opened 1 year ago

x4080 commented 1 year ago

from https://huggingface.co/blog/4bit-transformers-bitsandbytes ?

JetRunner commented 1 year ago

Yes, just modify the utils.py following the example from HF blog:

model = AutoModelForCausalLM.from_pretrained("facebook/opt-350m", load_in_4bit=True, device_map="auto")

We'll update the code after testing.

x4080 commented 1 year ago

cool, thanks @JetRunner