alxspiker
commented
1 year ago Script used for benchmarking: Requires llama-cpp-python==0.1.49
import json
import argparse
from llama_cpp import Llama
parser = argparse.ArgumentParser()
parser.add_argument("-m", "--model", type=str, default="./newggjt.bin")
args = parser.parse_args()
llm = Llama(model_path=args.model, n_threads=6)
stream = llm(
"Question: What are the names of the planets in the solar system? Answer: ",
max_tokens=48,
stop=["Q:", "\n"],
stream=True,
)
for output in stream:
print(output["choices"][0]["text"], end="")
#print(json.dumps(output, indent=2))
Max Threads = Poor Performance on 8 thread processor and GGJT model after convert.py
TL:DR - Try setting n_threads to 6 instead of 8 if you have an 8 thread processor. Getting consistently faster results than trying to use all of my 8 threads.Been doing some testing with a GGJT model to try to get the best performance on a little laptop. I did 2 tests for each change to n_threads. Tests were conducted while nothing else was open.Results On an 8 thread CPU
n_threads=1
Test 1
Test 2
n_threads=2
Test 1
Test 2
n_threads=4
Test 1
Test 2
n_threads=6
Test 1
Test 2
n_threads=7 (Seems better than 8, but not as good as 6)
Test 1
Test 2
n_threads=8 (Max threads)
Test 1
Test 2