Open kyle-v6x opened 1 month ago
I was able to reproduce the issue too:
Awaiting the assignment_ref
in the repro seems to also wait for the request to finish. Instead, it should return as soon as the request has been assigned to a replica.
This effectively prevents any Async task from being computed while previous tasks are running (i.e passing by ref).
Hi @kyle-v6x can you say more about this, and what you are trying to use the _to_object_ref()
API for?
@zcin In order to pass the result of a serve replica to an actor async, we need to retrieve the ObjectRef
and send it to the actor, as the actor has no concept of a DeploymentResponse
. Essentially the use case explained in the documentation here.
Further, if you want to run a task fully async (I know this is not supported officially yet) it makes sense to retrieve/return the task_id
for tracking. Since we can't obtain the ObjectRef
async, we can't get the task_id
.
I'll link this one more time for visibility. I may have time next week to finally build locally and work on a solution, but I can't guaruntee!
Hi @kyle-v6x, your use case makes sense to me, and you're correct that currently that isn't supported in Serve because the second object ref won't be retrieved until the request finishes. We are currently working on a fix!
Sorry, not resolved yet. This is depended on https://github.com/ray-project/ray/issues/46934
What happened + What you expected to happen
Since
2.10.0
, DeploymentResponses._to_object_ref() functions await the final result from the task, rather than returning the ObjectRef of the running task once it is scheduled. This effectively prevents any Async task from being computed while previous tasks are running (i.e passing by ref).You can find more details here.
Versions / Dependencies
This is an issue with
Ray[Serve]>=2.10.0
Reproduction script
Issue Severity
High: It blocks me from completing my task.