LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.
Apache License 2.0
2.54k
stars
198
forks
source link
Qwen-14B-INT8 face the issue: 'QwenTransformerLayerWeight' object has no attribute 'q_weight_' #333
Int the container by image: <ghcr.io/modeltc/lightllm:main> created, use below cmd:
start llm cmd:
process feedback below error:
gpu: