Serving in Kubernetes - Githubissues

InftyAI / llmaz-dashboard

A web console for llmaz.

https://github.com/InftyAI/llmaz

Apache License 2.0

3 stars 5 forks source link

Open kerthcet opened 12 months ago

kerthcet commented 12 months ago

First thoughts, we should have two different versions for serving:

Simple one, for demonstration or experimental, then we can create a deployment
Productization one, we'll deploy a ray cluster for auto-scaling, or even a operator if necessary.

kerthcet commented 12 months ago

This is also required when processing SFT.

kerthcet commented 12 months ago

/assign