search

MiuLab / Taiwan-LLM

Traditional Mandarin LLMs for Taiwan
https://twllm.com
Apache License 2.0
1.26k stars 104 forks source link

Support for AWQ quantization in TGI #59

Open nigue3025 opened 6 months ago

nigue3025 commented 6 months ago

Hi As I tried with 13b version in TGI, it works fine with bitsandbytes quantization. While trying with AWQ quantization in TGI, it shows error as "Cannot load 'awq' weight, make sure the model is already quantized" I am wondering if AWQ is too new to this model while deploying by TGI Or there is any suggestion or comment? Thanks

adamlin120 commented 6 months ago

For quantized model, i only tried with AWQ on vllm. you can find -awq model on my huggingface