Open changshivek opened 2 months ago
update: this error message also shows when using 0.5.0 version of vllm:
ERROR 06-13 02:36:52 async_llm_engine.py:532] Engine iteration timed out. This should never happen!
ERROR 06-13 02:36:52 async_llm_engine.py:52] Engine background task failed
ERROR 06-13 02:36:52 async_llm_engine.py:52] Traceback (most recent call last):
ERROR 06-13 02:36:52 async_llm_engine.py:52] File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 503, in engine_step
ERROR 06-13 02:36:52 async_llm_engine.py:52] request_outputs = await self.engine.step_async()
ERROR 06-13 02:36:52 async_llm_engine.py:52] File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 235, in step_async
ERROR 06-13 02:36:52 async_llm_engine.py:52] output = await self.model_executor.execute_model_async(
ERROR 06-13 02:36:52 async_llm_engine.py:52] File "/usr/local/lib/python3.10/dist-packages/vllm/executor/distributed_gpu_executor.py", line 166, in execute_model_async
ERROR 06-13 02:36:52 async_llm_engine.py:52] return await self._driver_execute_model_async(execute_model_req)
ERROR 06-13 02:36:52 async_llm_engine.py:52] File "/usr/local/lib/python3.10/dist-packages/vllm/executor/multiproc_gpu_executor.py", line 149, in _driver_execute_model_async
ERROR 06-13 02:36:52 async_llm_engine.py:52] return await self.driver_exec_model(execute_model_req)
ERROR 06-13 02:36:52 async_llm_engine.py:52] asyncio.exceptions.CancelledError
ERROR 06-13 02:36:52 async_llm_engine.py:52]
ERROR 06-13 02:36:52 async_llm_engine.py:52] During handling of the above exception, another exception occurred:
ERROR 06-13 02:36:52 async_llm_engine.py:52]
ERROR 06-13 02:36:52 async_llm_engine.py:52] Traceback (most recent call last):
ERROR 06-13 02:36:52 async_llm_engine.py:52] File "/usr/lib/python3.10/asyncio/tasks.py", line 456, in wait_for
ERROR 06-13 02:36:52 async_llm_engine.py:52] return fut.result()
ERROR 06-13 02:36:52 async_llm_engine.py:52] asyncio.exceptions.CancelledError
ERROR 06-13 02:36:52 async_llm_engine.py:52]
ERROR 06-13 02:36:52 async_llm_engine.py:52] The above exception was the direct cause of the following exception:
ERROR 06-13 02:36:52 async_llm_engine.py:52]
ERROR 06-13 02:36:52 async_llm_engine.py:52] Traceback (most recent call last):
ERROR 06-13 02:36:52 async_llm_engine.py:52] File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 42, in _log_task_completion
ERROR 06-13 02:36:52 async_llm_engine.py:52] return_value = task.result()
ERROR 06-13 02:36:52 async_llm_engine.py:52] File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 529, in run_engine_loop
ERROR 06-13 02:36:52 async_llm_engine.py:52] has_requests_in_progress = await asyncio.wait_for(
ERROR 06-13 02:36:52 async_llm_engine.py:52] File "/usr/lib/python3.10/asyncio/tasks.py", line 458, in wait_for
ERROR 06-13 02:36:52 async_llm_engine.py:52] raise exceptions.TimeoutError() from exc
ERROR 06-13 02:36:52 async_llm_engine.py:52] asyncio.exceptions.TimeoutError
ERROR:asyncio:Exception in callback functools.partial(<function _log_task_completion at 0x7fcdf0cc3ac0>, error_callback=<bound method AsyncLLMEngine._error_callback of <vllm.engine.async_llm_engine.AsyncLLMEngine object at 0x7fcea3705930>>)
handle: <Handle functools.partial(<function _log_task_completion at 0x7fcdf0cc3ac0>, error_callback=<bound method AsyncLLMEngine._error_callback of <vllm.engine.async_llm_engine.AsyncLLMEngine object at 0x7fcea3705930>>)>
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 503, in engine_step
request_outputs = await self.engine.step_async()
File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 235, in step_async
output = await self.model_executor.execute_model_async(
File "/usr/local/lib/python3.10/dist-packages/vllm/executor/distributed_gpu_executor.py", line 166, in execute_model_async
return await self._driver_execute_model_async(execute_model_req)
File "/usr/local/lib/python3.10/dist-packages/vllm/executor/multiproc_gpu_executor.py", line 149, in _driver_execute_model_async
return await self.driver_exec_model(execute_model_req)
asyncio.exceptions.CancelledError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.10/asyncio/tasks.py", line 456, in wait_for
return fut.result()
asyncio.exceptions.CancelledError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 42, in _log_task_completion
return_value = task.result()
File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 529, in run_engine_loop
has_requests_in_progress = await asyncio.wait_for(
File "/usr/lib/python3.10/asyncio/tasks.py", line 458, in wait_for
raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "uvloop/cbhandles.pyx", line 63, in uvloop.loop.Handle._run
File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 54, in _log_task_completion
INFO 06-13 02:36:52 async_llm_engine.py:167] Aborted request cmpl-5b8a3cb3d04d41babc4903e537e39544.
raise AsyncEngineDeadError(
vllm.engine.async_llm_engine.AsyncEngineDeadError: Task finished unexpectedly. This should never happen! Please open an issue on Github. See stack trace above for theactual cause.
How do you solve this problem, I have the same problem as you
Ref. #5732
How do you solve this problem, I have the same problem as you
I'm looking into the AsyncEngineDeadError
issues under #5901
Could you please share the following so that I can reproduce the issue:
I launched vllm inference server on kubernets, here is my launch command
containers:
- name: qw2-vllm #自定义名字
image: harbor.myharbor.cn/myrepo/vllm-openai:v0.5.0 #推理镜像的名字
ports:
- containerPort: 8000
# securityContext:
# privileged: true
# command: ["/bin/sh","-c", "sleep 1d"] #推理启动的脚本
command: ["/bin/bash", "-c"] #推理启动的脚本
args: [
# "sudo sed -i '175,+2s/\"dns.google\"/\"8.8.8.8\"/g'
/workspace/vllm/utils.py && \
"nvidia-smi;python3 -m vllm.entrypoints.openai.api_server \
--host 0.0.0.0 \
--model /fl/nlp/common/plms/qwen2/Qwen2-72B-Instruct \
--trust-remote-code \
--enforce-eager \
--max-model-len 32768 \
--gpu-memory-utilization 0.9 \
--served-model-name qwen2-72bc \
--tensor-parallel-size 8"
]
resources:
limits:
nvidia.com/gpu: 8 #必须填写,每个节点使用的 GPU 的数量
since this bug seems not being triggered by specific cases, I am unable to provide such cases, sorry
Robert Shaw @.***> 于2024年6月27日周四 19:54写道:
I'm looking into the AsyncEngineDeadError issues under #5901 https://github.com/vllm-project/vllm/issues/5901
Could you please share the following so that I can reproduce the issue:
- The server launch command used
- A sample request that causes the error
— Reply to this email directly, view it on GitHub https://github.com/vllm-project/vllm/issues/5443#issuecomment-2194482043, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFST5AJGBRLECCBNDHC3V3DZJP4PHAVCNFSM6AAAAABJFZKTTKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJUGQ4DEMBUGM . You are receiving this because you authored the thread.Message ID: @.***>
Your current environment
🐛 Describe the bug
I use vllm/vllm-openai:v0.4.3 on k8s to deploy a 14b model with config as follows:
after a short period of normal run, this happened:
I saw the exception message said "This should never happen" but this happened. Anyone with the same issue? Is there any solutions for this?