search

substratusai / kubeai

Private Open AI on Kubernetes
https://www.kubeai.org
Apache License 2.0
362 stars 33 forks source link

Expose basic Autoscaling metrics #199

Open samos123 opened 3 weeks ago

samos123 commented 3 weeks ago

autoscaling metrics.. things like number of qps per replica, number of replicas, autoscaling time (e.g. time from pod request to pod ready)

nstogner commented 3 weeks ago

Note: KubeAI already exposes a metrics endpoint with some boring metrics. We just need to add these to that endpoint.

NOTE: Alex made a contribution to Lingo that can be used as a reference: https://github.com/substratusai/kubeai/commit/b476ca268f1cf568490cabafb0201b22c8a99bfb

nstogner commented 6 days ago

We should do this with OpenTelemetry as that appears to be the way everything is heading.

alpe commented 6 days ago

👍 if this is not super urgent, I would work on this later this week.

nstogner commented 6 days ago

Hey @alpe - I’ve started on this one already. I forgot to assign it. Would love a review from you though!