edoakes
commented
2 years ago Hey @klwuibm this use case looks really exciting and thanks for the detailed feature request!
This is something we've thought about in the past, but didn't have any users with strong requirements for it yet. I think getting a basic version of this off the ground would be straightforward, but a "complete" solution would be a decent chunk of work. Specifically, handling scaling up/scaling down the number of replicas would require some further thought because we need some way to repartition/can no longer simply hash and shard across the replicas. Handling rolling upgrades also requires further thought.
For a basic version that doesn't handle scaling or upgrading though, I think all we'd need to do is the following:
- Add an option to the
ServeHandlethat takes a user-provided ID to be used to map to a replica. - Ensure that the mapping of hash -> replica is consistent. We would need a way to sort the replicas (could just be done by replica name) and ensure that this is consistent across replica restarts. Right now if a replica crashes, we start a new one with a new name in its place. We'd need to change this to instead restart the same replica with the same name/id.
How urgent is this for your use case? We are pretty resource-constrained right now so I'm not sure when we'd be able to prioritize this feature. We would be happy to help with a design & shepherd a contribution if you are willing/able to work on it!
ericl
klwuibm
yuanchi2807
Search before asking
Description
As a data scientist, I would like to use Ray Serve to deploy stateful Ray workflows to take advantage of auto-scaling of Ray Serve deployment. When a stateful workflow is deployed, the workflow's state may change as incoming requests are processed by the workflow. Hence, the incoming requests with the same key/ID should be routed to the same Ray Serve replica that executes the workflow. Namely, the routing of incoming requests to the Ray Serve replicas should be sticky and should be based on a key/ID.
However, the current routing strategy of Ray Serve for incoming requests to a replica is round-robin, i.e., an incoming request can be routed to any Serve replica.
Use case
As an example, assume a Ray Workflow has been deployed via Ray Serve for a fraudulent credit card transaction detection application. Each workflow is specific to a card number and holds a state that changes as new transactions are continuously processed. New transactions are sent to the deployment for flagging potential fraudulent transactions. They should be routed to the proper Serve replicas running workflows with specific card numbers. This type of applications requires low latency and auto-scaling, as the incoming requests can be bursty.
Success metric: implementing sticky sessions should see reduced response latency and increased concurrent workflow throughput.
Related issues
One potential way to implement this feature is to associate a key/ID with a request, and use a hash function to route the request to a Ray Serve replica. Since there tends to be a lot more different keys/IDs from the incoming requests than the number of Ray Serve replicas, load balancing among the replicas needs to be addressed.
Additionally, a Ray Serve replica is likely to process requests coming to workflows with different keys/IDs, some of the workflow states may be overflown out of the main memory and saved to disks. As a result, the workflow states might have to be loaded back from disks when they are needed to process a new incoming request.
Are you willing to submit a PR?