Closed rtwang1997 closed 7 months ago
Could you provide a minimal repro for your setup? Do the vLLM workers die randomly, or is there a pattern?
Hi,
We are finding that the vLLM workers die after running for around 1 hour (give or take), consistently.
We tried with a very simple ray service, where a deployment creates an actor in it's init function and I get the same behaviour where the actor dies after ~1h because all reference to it get removed. Here's the sample code:
import ray
from ray import serve
from fastapi import FastAPI
app = FastAPI()
@ray.remote
class Actor:
def __call__(self):
return "hello world"
@serve.deployment
@serve.ingress(app)
class APIIngress:
def __init__(self):
self.actor = Actor.remote()
@app.get("/health")
async def healthcheck(self):
"""
checks if server is up.
"""
return {"status": "server up"}
@app.get("/test")
async def test(self):
return await self.actor.remote()
SERVE_APP = APIIngress.bind()
Thanks for the repro! What Ray version are you using?
Hi,
We are using Ray version 2.7.1
Are there any updates here @rtwang1997? I synced with one of your coworkers on Slack and proposed some approaches, but I'm not sure how it went.
@rtwang1997 I'll close this issue for now. Feel free to reopen if you're still running into the problem.
Hi, we are running into a weird issue where the Ray Workers created by VLLM are being killed, even though the deployment itself stays alive. The effect of this is that when you make a request to a model deployment, the following error message occurs, since the workers are already dead but
AviaryLLMEngine
still has a reference to them via theworkers
object property:In the logs, it appears that the Ray Workers created by VLLM are being killed by GCS with the following message:
More specifically, this part of the message:
The actor is dead because all references to the actor were removed
appears to indicate that GCS is killing the Ray Workers because it believes that there are no references left to the Ray Workers from anywhere.However, I don't understand how this could be the case, since the top-level Ray Serve deployment is still alive and the deployment holds on to the
VLLMEngine
as a reference.VLLMEngine
holds on toAviaryAsyncLLMEngine
asself.engine
, and theAviaryAsyncLLMEngine
holds on toAviaryLLMEngine
which has a reference to the workers asself.workers
.If the top-level deployment hasn't died, I don't see how the reference count on the workers could have been decremented/why GCS would think that these actors are out of scope.