search

ray-project / ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
https://ray.io
Apache License 2.0
34.16k stars 5.8k forks source link

[Serve] Observability for proxy #48184

Open smit-kiri opened 1 month ago

smit-kiri commented 1 month ago

Description

It would be great to have an observability metric for the time it takes for the request from entering the HTTP Proxy and entering the replica.

Use case

This will be very helpful to debug performance issues and narrow down on the component that may be causing slowdowns.

jcotant1 commented 2 weeks ago

Cc: @alanwguo @zcin

akshay-anyscale commented 5 days ago

Thanks for creating the ticket @smit-kiri , are you open to contributing this?