Open KiwiHana opened 10 months ago
Hi @rnwang04 could you help on this issue?
Hi @KiwiHana , I test this case on my local arc machine and I can't reproduce your results. ChatGLM2-6B bigdl-llm[xpu] 20231101 on Arc 770 with i9-12900K. env setting:
export USE_XETLA=OFF
export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1
For 32in/32out,instance=1, rest latency is 20.1ms/token. For 32in/32out,instance=2, rest latency is 32.4ms/token.
Would you mind testing it without tcmalloc again? Or maybe this dismatch is caused by CPU freq?
ChatGLM2-6B use multi-batch size by bigdl-llm[xpu] 20231016 on Arc 770 with Xeon.
For 32in/32out,instance=1, rest latency is 20.5ms/token. For 32in/32out,instance=2, rest latency is 224.5ms/token.
download script: https://github.com/biyuehuang/LLM_Arc_UI/blob/main/gpu_benchmark/test_chatglm2.py