支持的量化类型 - Githubissues

SJTU-IPADS / PowerInfer

High-speed Large Language Model Serving on PCs with Consumer-grade GPUs

MIT License

7.98k stars 415 forks source link

Closed deleteeeee closed 1 month ago

deleteeeee commented 5 months ago

您好，请问一下，目前只支持F16和Q4_0这两种么，我在运行其它量化类型的的模型时会出现unsupported type

Dujianhua1008 commented 5 months ago

？？？原来是这里的问题我做了 Q8 Q4_k 跑起来都是 ###