Closed KramPiotr closed 6 months ago
run faster not slower
(I don't have experience about running time and llama.cpp internals) Usually you want to send more than just 1 layer - like as many as possible - to the gpu for faster times. But in the end its a valid configuration. Be aware the first call also boots up the model so some time is lost there as well. Can you provide any kind of time benchmark between those options? E.g. Prepend the time
command to your curl call and/or the log output regarding `stderr print_timings:``?
or erroneous
Once this case happened, whats the output of curl and the local-ai log? Does it throw anything else than 200 - POST /v1/completions
? Did you make sure that you don't run into your curl's default timeout?
TLDR:
I was interested how it looked on my machine:
Details:
docker:
docker run -d --name localai -p 8080:8080 -v models:/models --gpus all -e MODELS_PATH=/models -e DEBUG=true -e THREADS=6 quay.io/go-skynet/local-ai:master-cublas-cuda12-ffmpeg-cor
script:
#!/bin/bash
echo
echo "restart localai container"
docker restart localai
sleep 5
echo
echo "warmup cpu"
curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "mistral-test-cpu",
"messages": [{"role": "user", "content": "Hello! Give only a short answer."}]
}' >/dev/null 2>&1
echo
echo "cpu"
time curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "mistral-test-cpu",
"messages": [{"role": "assistant", "content": "How old is Mickey Mouse?"}]
}'
echo
echo "restart localai container"
docker restart localai
sleep 5
echo
echo "warmup 1layer"
curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "mistral-test-1layer",
"messages": [{"role": "user", "content": "Hello! Give only a short answer."}]
}' >/dev/null 2>&1
echo
echo "1layer"
time curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "mistral-test-1layer",
"messages": [{"role": "assistant", "content": "How old is Mickey Mouse?"}]
}'
echo
echo "restart localai container"
docker restart localai
sleep 5
echo
echo "warmup gpu"
curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "mistral-test-gpu",
"messages": [{"role": "user", "content": "Hello! Give only a short answer."}]
}' >/dev/null 2>&1
echo
echo "gpu"
time curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "mistral-test-gpu",
"messages": [{"role": "assistant", "content": "How old is Mickey Mouse?"}]
}'
output:
bash test.sh
restart localai container
localai
warmup cpu
cpu
{"created":1704887210,"object":"chat.completion","id":"09166843-6cb6-4e6e-874a-525f0aaa47a8","model":"mistral-test-cpu","choices":[{"index":0,"finish_reason":"stop","message":{"role":"assistant","content":"\n Mickey Mouse, a fictional character created in 1928, does not have an age in the traditional sense. However, if we were to consider Mickey's age in terms of
the character's existence, he would be 93 years old as of 2021."}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}
real 0m5.365s
user 0m0.003s
sys 0m0.000s
restart localai container
localai
warmup 1layer
1layer
{"created":1704887237,"object":"chat.completion","id":"2d4ac21a-c2ba-4e4f-a5bb-ef3e5c724487","model":"mistral-test-1layer","choices":[{"index":0,"finish_reason":"stop","message":{"role":"assistant","content":"\n Mickey Mouse, a fictional character created in 1928, does not have an age in the traditional sense. However, if we were to consider Mickey's age in terms
of the character's existence, he would be 93 years old as of 2021."}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}
real 0m5.847s
user 0m0.003s
sys 0m0.000s
restart localai container
localai
warmup gpu
gpu
{"created":1704887264,"object":"chat.completion","id":"2969ee7a-5ecb-45c1-a015-2b7af618391e","model":"mistral-test-gpu","choices":[{"index":0,"finish_reason":"stop","message":{"role":"assistant","content":"\n Mickey Mouse, a fictional character created in 1928, does not have an age in the traditional sense. However, if we were to consider Mickey's age in terms of
the character's existence, he would be 93 years old as of 2021."}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}
real 0m1.428s
user 0m0.003s
sys 0m0.000s
configs (separate in their respective files):
context_size: 1024
name: mistral-test-cpu
parameters:
model: mistral-7b-openorca.Q3_K_S.gguf
temperature: 0.2
top_k: 80
top_p: 0.7
template:
chat: chat
chat_message: chatml
completion: completion
threads: 6
gpu_layers: 0
---
context_size: 1024
name: mistral-test-1layer
parameters:
model: mistral-7b-openorca.Q3_K_S.gguf
temperature: 0.2
top_k: 80
top_p: 0.7
template:
chat: chat
chat_message: chatml
completion: completion
threads: 6
gpu_layers: 1
---
context_size: 1024
name: mistral-test-gpu
parameters:
model: mistral-7b-openorca.Q3_K_S.gguf
temperature: 0.2
top_k: 80
top_p: 0.7
template:
chat: chat
chat_message: chatml
completion: completion
threads: 6
gpu_layers: 100
cpu:
11:30AM DBG GRPC(mistral-7b-openorca.Q3_K_S.gguf-127.0.0.1:34563): stderr llm_load_tensors: using CUDA for GPU acceleration
11:30AM DBG GRPC(mistral-7b-openorca.Q3_K_S.gguf-127.0.0.1:34563): stderr WARNING: failed to allocate 3017.28 MB of pinned memory: out of memory
11:30AM DBG GRPC(mistral-7b-openorca.Q3_K_S.gguf-127.0.0.1:34563): stderr llm_load_tensors: system memory used = 3017.39 MiB
11:30AM DBG GRPC(mistral-7b-openorca.Q3_K_S.gguf-127.0.0.1:34563): stderr llm_load_tensors: offloading 0 repeating layers to GPU
11:30AM DBG GRPC(mistral-7b-openorca.Q3_K_S.gguf-127.0.0.1:34563): stderr llm_load_tensors: offloaded 0/33 layers to GPU
...
11:31AM DBG GRPC(mistral-7b-openorca.Q3_K_S.gguf-127.0.0.1:34563): stderr print_timings: total time = 5358.12 ms
1layer:
11:31AM DBG GRPC(mistral-7b-openorca.Q3_K_S.gguf-127.0.0.1:39319): stderr llm_load_tensors: using CUDA for GPU acceleration
11:31AM DBG GRPC(mistral-7b-openorca.Q3_K_S.gguf-127.0.0.1:39319): stderr WARNING: failed to allocate 2927.87 MB of pinned memory: out of memory
11:31AM DBG GRPC(mistral-7b-openorca.Q3_K_S.gguf-127.0.0.1:39319): stderr llm_load_tensors: system memory used = 2927.98 MiB
11:31AM DBG GRPC(mistral-7b-openorca.Q3_K_S.gguf-127.0.0.1:39319): stderr llm_load_tensors: VRAM used = 89.41 MiB
11:31AM DBG GRPC(mistral-7b-openorca.Q3_K_S.gguf-127.0.0.1:39319): stderr llm_load_tensors: offloading 1 repeating layers to GPU
11:31AM DBG GRPC(mistral-7b-openorca.Q3_K_S.gguf-127.0.0.1:39319): stderr llm_load_tensors: offloaded 1/33 layers to GPU
...
11:32AM DBG GRPC(mistral-7b-openorca.Q3_K_S.gguf-127.0.0.1:39319): stderr print_timings: total time = 5840.45 ms
gpu:
11:32AM DBG GRPC(mistral-7b-openorca.Q3_K_S.gguf-127.0.0.1:35869): stderr llm_load_tensors: using CUDA for GPU acceleration
11:32AM DBG GRPC(mistral-7b-openorca.Q3_K_S.gguf-127.0.0.1:35869): stderr llm_load_tensors: system memory used = 53.83 MiB
11:32AM DBG GRPC(mistral-7b-openorca.Q3_K_S.gguf-127.0.0.1:35869): stderr llm_load_tensors: VRAM used = 2963.56 MiB
11:32AM DBG GRPC(mistral-7b-openorca.Q3_K_S.gguf-127.0.0.1:35869): stderr llm_load_tensors: offloading 32 repeating layers to GPU
11:32AM DBG GRPC(mistral-7b-openorca.Q3_K_S.gguf-127.0.0.1:35869): stderr llm_load_tensors: offloading non-repeating layers to GPU
11:32AM DBG GRPC(mistral-7b-openorca.Q3_K_S.gguf-127.0.0.1:35869): stderr llm_load_tensors: offloaded 33/33 layers to GPU
...
11:32AM DBG GRPC(mistral-7b-openorca.Q3_K_S.gguf-127.0.0.1:35869): stderr print_timings: total time = 1421.91 ms
LocalAI version:
I am on commit: 574fa67bdcafd618859fcda4d239f10f326182a6
Environment, CPU architecture, OS, and Version:
I am on Windows and using WSL2 with Ubuntu 22.04:
Linux LAPTOP-FN76EJ99 5.15.133.1-microsoft-standard-WSL2 #1 SMP Thu Oct 5 21:02:42 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Describe the bug
Running llama-2 with the gpu_layers param commented out (no GPU offloading) works but with 1+ GPU layers offloaded the requests comes up with an error or runs much slower (I haven't seen it succeded in this scenario yet). Querying the model with no GPU layers param completes the basic request in a few minutes.
To Reproduce
The command I use to run Local AI:
docker run --rm -ti --env-file .env --gpus all -p 8080:8080 -v $(pwd)/models:/models quay.io/go-skynet/local-ai:master-cublas-cuda12
.env:
Configuration for the model I am using (llama-2-13b.yaml):
The command I use for testing:
The output of docker run when no GPU layers set (that also contains my GPU information):
The lines that change when I set the GPU layers:
nvidia-smi output when running the inference with 1 GPU layer offloaded:
Expected behavior
GPU utilisation should make the model run faster not slower or erroneous.