Closed forcekeng closed 7 months ago
Hi, we test the same code and Misral-7B-Instruct-v0.2 model in i9-13900K and 64GB memory, the inference time is 17.881616592407227s. For Llama-2-7b it's 19.197434425354004s.
And we test llm-cli for Llama-2-7b using same prompt and max-output-token in same i9 machine, the time cost is 14.29773s.
Maybe you can check your bigdl-llm version, or try it in a docker container and check if there any difference in environment variables between container and your environment. I used image intelanalytics/bigdl-llm-cpu:2.5.0-SNAPSHOT for testing.
And we test llm-cli for Llama-2-7b using same prompt and max-output-token in same i9 machine, the time cost is 14.29773s.
OK, thanks for your reply. I'll check it and compare the python code with llm-cli
and llama.cpp
.
I try the code at file
python/llm/example/CPU/HF-Transformers-AutoModels/Model/mistral/generate.py
, there's no acceleration on my CPU. I try it on Windows and Windows-WSL both. But both thellm-cli
and originalllama.cpp
speeds up the model about 10x faster on my CPU. Is there anyone know how the given code could run faster?Model tested:
llama_7b
,llama_7b_chat
,mistral_7b_intruct_v0.2
. CPU: 12th Gen Intel(R) Core(TM) i7-12700KF Memory: 32GTest code:
The output: