RTP-LLM 模式下，llama3.1 FP16 效果不一样

alibaba / rtp-llm

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

Apache License 2.0

521 stars 48 forks source link

Open anigi98932 opened 2 weeks ago

anigi98932 commented 2 weeks ago

使用huggingface载入 llama3.1的生成结果，和RTP-LLM 载入结果不一致同样的prompt，无法重现在RTP-LLM