P40使用moss-moon-003-sft-int4/量化模型推理报错

OpenMOSS / MOSS

An open-source tool-augmented conversational language model from Fudan University

https://txsun1997.github.io/blogs/moss.html

Apache License 2.0

11.9k stars 1.14k forks source link

P40使用moss-moon-003-sft-int4/量化模型推理报错 #302

Closed dubaoquan404 closed 1 year ago

dubaoquan404 commented 1 year ago

python3: /project/lib/Analysis/Utility.cpp:136: bool mlir::supportMMA(mlir::Value, int): Assertion `(version == 1 || version == 2) && "Unexpected MMA layout version found"' failed.

13875892872 commented 1 year ago

我也是同样的错误，P40 int4量化推理报错。

boystray commented 1 year ago

Pascal架构的显卡不支持，硬件问题。

boystray commented 1 year ago

Pascal架构的显卡不支持，硬件问题。

https://github.com/qwopqwop200/GPTQ-for-LLaMa/issues/142

boystray commented 1 year ago

试试这个方案 https://github.com/openai/triton/issues/1271