Wolfsauge
commented
4 months ago I retried after terminating all containers using the GPU, resulting in the end of the process with PID 18761.
❯ nvidia-smi
Wed May 1 09:10:23 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.76 Driver Version: 550.76 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 3090 Off | 00000000:01:00.0 Off | N/A |
| 0% 32C P8 21W / 420W | 18452MiB / 24576MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 41908 C python3 18438MiB |
+-----------------------------------------------------------------------------------------+The behavior remains the same.
~/Desktop/GitHub
❯ time curl -s http://mnemosyne.local:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Meta-Llama-3-8B-Instruct", "n": 22, "temperature": 0.3, "min_p": 0.1, "max_tokens": "7168",
"messages": [
{
"role": "user",
"content": "Tell me a long story about llamas. Then analyze the story, focusing on the stylistic devices."
}
]
}' > response-batch-22-sampling.json
curl -s http://mnemosyne.local:8000/v1/chat/completions -H -d > 0.01s user 0.01s system 0% cpu 36.651 totalWhile the request with "n": 22 works again, the following request fails.
~/Desktop/GitHub 36s
❯ time curl -s http://mnemosyne.local:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Meta-Llama-3-8B-Instruct", "n": 23, "temperature": 0.3, "min_p": 0.1, "max_tokens": "7168",
"messages": [
{
"role": "user",
"content": "Tell me a long story about llamas. Then analyze the story, focusing on the stylistic devices."
}
]
}' > response-batch-23-sampling.jsonFrom here the request with "n": 23 hangs. The engine becomes unresponsive, does not log anything suspicious and needs a restart. New requests are accepted, but will not cause any text to be generated.
robertgshaw2-neuralmagic
Your current environment
🐛 Describe the bug
When making a web request to the OpenAI-compatible API front end running from the official docker image like shown below, this will crash the engine each time a request is made.
On my setup the below request always succeeds with "n": 22, while it will always fail with "n": 23.
The failing request will not terminate, the GPU compute visible in nvtop will go down. After this happens, new requests will not be served by the engine and also hang indefinitely. Once a failing request was made, the frequent log entries of vllm continue appearing in the log. However, text generation stops and can only be restarted by replacing the container.
Because of this, I decided to open a bug report.
The model used to reproduce: meta-llama/Meta-Llama-3-8B-Instruct
Command used to start the engine:
Succeeding request
Failing request
Note: while this is happening, another process is living in the 20% free GPU VRAM.
Thank you!