search

opendatahub-io / caikit-tgis-serving

Apache License 2.0
17 stars 39 forks source link

[RFE] Add openai text generation API compatibility layer in kserve/TGIS #226

Open ashishkamra opened 5 months ago

ashishkamra commented 5 months ago

Request The ask is to introduce a openai text generation API compatibility layer (chat completion endpoint) to kserve/TGIS.

Why Having an openai API compatibility layer will allow more open source tools such as 'https://github.com/EleutherAI/lm-evaluation-harness' to interoperate with our model serving stack

Suggested Implemention Use litellm openai proxy server - https://litellm.vercel.app/docs/providers/huggingface

danielezonca commented 5 months ago

One idea to explore is to use a similar proxy as KServe transformer around TGIS image so that we can easily plug it in the ServingRuntime. The existing HF provided will not probably work OOTB with TGIS but the work to adapt should not be too much.

@Xaenalt @dtrifiro Wdyt? (I'm just thinking for now at a community/research spike, nothing long term/supported)

danielezonca commented 5 months ago

FYI KServe project itself is considering to add OpenAI API support for chat completition as part of open-inference-protocol spec