Support for loading in lower precision

TsinghuaDatabaseGroup / DB-GPT

An LLM Based Diagnosis System (https://arxiv.org/pdf/2312.01454.pdf)

http://dbgpt.dbmind.cn/

Apache License 2.0

565 stars 80 forks source link

Closed JINO-ROHIT closed 10 months ago

JINO-ROHIT commented 11 months ago

i have a small gpu, can i load the 13b llama in 4bit? Is there support for this?

zhouxh19 commented 11 months ago

we will respond to this question after well testing in 4bit.

JINO-ROHIT commented 11 months ago

sure thank you

curtis-sun commented 10 months ago

We have provided a 4bit version (https://huggingface.co/curtis-sun/diag-baichuan2-4bit/tree/main) fine-tuned from Baichuan2-13B. Later we'll also upload more fine-tuned models.