Load-balancing / auto-scaling for LLM serving on Azure

anarchy-ai / LLM-VM

irresponsible innovation. Try now at https://chat.dev/

https://anarchy.ai/

MIT License

477 stars 148 forks source link

Load-balancing / auto-scaling for LLM serving on Azure #376

Open VictorOdede opened 11 months ago

internot169 commented 9 months ago

Hi @VictorOdede, I'd like to try this issue. Could you please provide some more information?

kaushikdaiv7 commented 8 months ago

I would love to work on this @VictorOdede , can you share details for this issue

lucylililiwang commented 7 months ago

Hi! Vik, Can I also please work on this issue? Thank you!

lucylililiwang commented 7 months ago

When we are taking care of the Load-balancing, is it alright for us to do Azure Kubernetes Service (AKS) along with Horizontal Pod Autoscaler (HPA) and Kubernetes Ingress Controller for load balancing? Thank you!

lucylililiwang commented 7 months ago

sorry, when we setting up the Kubernetes cluster, which Kubernetes version should we choose? Thank you! kubernates