Open Wolfsauge opened 4 months ago
I retried after terminating all containers using the GPU, resulting in the end of the process with PID 18761.
❯ nvidia-smi
Wed May 1 09:10:23 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.76 Driver Version: 550.76 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 3090 Off | 00000000:01:00.0 Off | N/A |
| 0% 32C P8 21W / 420W | 18452MiB / 24576MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 41908 C python3 18438MiB |
+-----------------------------------------------------------------------------------------+
The behavior remains the same.
~/Desktop/GitHub
❯ time curl -s http://mnemosyne.local:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Meta-Llama-3-8B-Instruct", "n": 22, "temperature": 0.3, "min_p": 0.1, "max_tokens": "7168",
"messages": [
{
"role": "user",
"content": "Tell me a long story about llamas. Then analyze the story, focusing on the stylistic devices."
}
]
}' > response-batch-22-sampling.json
curl -s http://mnemosyne.local:8000/v1/chat/completions -H -d > 0.01s user 0.01s system 0% cpu 36.651 total
While the request with "n": 22 works again, the following request fails.
~/Desktop/GitHub 36s
❯ time curl -s http://mnemosyne.local:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Meta-Llama-3-8B-Instruct", "n": 23, "temperature": 0.3, "min_p": 0.1, "max_tokens": "7168",
"messages": [
{
"role": "user",
"content": "Tell me a long story about llamas. Then analyze the story, focusing on the stylistic devices."
}
]
}' > response-batch-23-sampling.json
From here the request with "n": 23 hangs. The engine becomes unresponsive, does not log anything suspicious and needs a restart. New requests are accepted, but will not cause any text to be generated.
Thanks for reporting. I managed to recreate the error.
What is happening is that this request is generating so many tokens that we run out of memory in the KV cache!
When this happens we swap the KVs to CPU RAM to wait until more GPU memory frees up to accomodate the request. But, since a single request fills up the entire KV cache, this will never happen. Thus, request is "stuck" in the SWAPPED
state permanently.
INFO 05-01 12:12:13 metrics.py:229] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 1 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 71.9%
We will discuss how we want to handle this case
cc @zhuohan123 @simon-mo
Thank you for your reproduction and feedback.
Your current environment
🐛 Describe the bug
When making a web request to the OpenAI-compatible API front end running from the official docker image like shown below, this will crash the engine each time a request is made.
On my setup the below request always succeeds with "n": 22, while it will always fail with "n": 23.
The failing request will not terminate, the GPU compute visible in nvtop will go down. After this happens, new requests will not be served by the engine and also hang indefinitely. Once a failing request was made, the frequent log entries of vllm continue appearing in the log. However, text generation stops and can only be restarted by replacing the container.
Because of this, I decided to open a bug report.
The model used to reproduce: meta-llama/Meta-Llama-3-8B-Instruct
Command used to start the engine:
Succeeding request
Failing request
Note: while this is happening, another process is living in the 20% free GPU VRAM.
Thank you!