search

project-baize / baize-chatbot

Let ChatGPT teach your own chatbot in hours with a single GPU!
https://arxiv.org/abs/2304.01196
GNU General Public License v3.0
3.16k stars 282 forks source link

How to access int8 with 13b version? #15

Open Meathelix1 opened 1 year ago

Meathelix1 commented 1 year ago

Hello, I would like to know how we can access the 13b int8 version for the demo?

JetRunner commented 1 year ago

https://github.com/project-baize/baize/blob/ccf0bb8485657b7c16a57456bbb835503bac2456/demo/app.py#L18

Change this line to tokenizer, model, device = load_tokenizer_and_model(base_model, adapter_model, load_8bit=True)