Support TGI as another alternative backend

InftyAI / llmaz

☸️ Easy, advanced inference platform for large language models on Kubernetes

Apache License 2.0

13 stars 5 forks source link

Support TGI as another alternative backend #72

Open kerthcet opened 1 month ago

kerthcet commented 1 month ago

What would you like to be added:

TGI is also a popular inference backend that we should support. More importantly, we should set this as an example once people would like to support another backend in the future.

Why is this needed:

Adopte more backends and make this a reference.

Completion requirements:

This enhancement requires the following artifacts:

[ ] Design doc
[ ] API change
[ ] Docs update

The artifacts should be linked in subsequent comments.

kerthcet commented 1 month ago

/kind feature

kerthcet commented 6 days ago

/milestone v0.1.0

kerthcet commented 1 day ago

A follow up of https://github.com/InftyAI/llmaz/issues/134