[Serve] design and implement transparent HTTP2 proxy

ray-project / ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

https://ray.io

Apache License 2.0

34.22k stars 5.81k forks source link

[Serve] design and implement transparent HTTP2 proxy #47370

Open GeneDer opened 3 months ago

GeneDer commented 3 months ago

Currently we are using uvicorn to implement http proxy. We need to find an HTTP2 compatible server to to replace it and implement http and grpc proxy on top of it.

Maybe try hypercorn

This will allow us to not relying on gRPC proxy's server implementation and transparently passing payload between the replica and the proxy.

Superskyyy commented 2 months ago

https://github.com/ray-project/ray/issues/46174