search

pcg-mlp / KsanaLLM

Other
286 stars 30 forks source link

No module named 'libtorch_serving' #5

Closed LiweiPE closed 5 months ago

LiweiPE commented 5 months ago

After installed buy cmake I got this error. What is the reason and how to fix it? image

xingjinglu commented 5 months ago

libtorch_serving.so 在你build的路径下面，例如 KsanaLLM/build/lib

需要将该路径设置在PYTHONPATH环境变量里， export PYTHONPATH=/yourpath/KsanaLLM/build/lib:$PYTHONPATH

LiweiPE commented 5 months ago

Thanks for the answer, I installed by distribute method and I got this error now: image

xingjinglu commented 5 months ago

I have not re-produce your error now.

For investigating this problem, please paste the error log here and give a description of your envritonment.

BTW, you build KsanaLLM with the nvidia image?

And give my environment for reference. CUDA: 11.8 Driver: 525.116.04 GPU: A10

LiweiPE commented 5 months ago

With nvidia image there is no error for installation. These errors come from conda environment installation. Final question, do you plan to support GPTQ quantization models in this framework? This issue can be closed, thanks.

pcg-mlp commented 5 months ago

GPTQ may be supported in Q3/Q4 2024. Scheduling performance and Ascend kernel optimization are in first priority right now