chore: Better horizontal and vertical scaling in the cluster

defenseunicorns / leapfrogai

Production-ready Generative AI for local, cloud native, airgap, and edge deployments.

https://leapfrog.ai

Apache License 2.0

250 stars 28 forks source link

chore: Better horizontal and vertical scaling in the cluster #816

Open CollectiveUnicorn opened 1 month ago

CollectiveUnicorn commented 1 month ago

Describe what should be investigated or refactored

Refactor LeapfrogAI's components and helm charts so that they can effectively horizontally and vertically scale to handle ~500 concurrent users. More advanced load balancing or changes to individual applications may also be needed.

CollectiveUnicorn commented 1 month ago

Needs additional work on what the implementation of this looks like