Serving in Kubernetes - Githubissues

InftyAI / Llmaz

⚒ A building block for users to build/serve their own LLM from A to Z on Kubernetes.

https://InftyAI.com

Apache License 2.0

3 stars 4 forks source link

Open kerthcet opened 7 months ago

kerthcet commented 7 months ago

First thoughts, we should have two different versions for serving:

Simple one, for demonstration or experimental, then we can create a deployment
Productization one, we'll deploy a ray cluster for auto-scaling, or even a operator if necessary.

kerthcet commented 7 months ago

This is also required when processing SFT.

kerthcet commented 7 months ago

/assign