Load-balancing / auto-scaling for LLM serving on AWS

anarchy-ai / LLM-VM

irresponsible innovation. Try now at https://chat.dev/

https://anarchy.ai/

MIT License

465 stars 150 forks source link

Open VictorOdede opened 8 months ago

horahoradev commented 8 months ago

What did you have in mind here? when you refer to autoscaling, are you referring to horizontal scalability, or parallelism across a single host?

VictorOdede commented 8 months ago

@horahoradev I was actually referring to horizontal scaling of LLM instances

lucylililiwang commented 4 months ago

Hi, can I please work on this issue? Thank you!