Any chance to support text-generation-inference as backend?

hyperonym / basaran

Basaran is an open-source alternative to the OpenAI text completion API. It provides a compatible streaming API for your Hugging Face Transformers-based text generation models.

MIT License

1.29k stars 80 forks source link

Any chance to support text-generation-inference as backend? #147

Open hewr1993 opened 1 year ago

hewr1993 commented 1 year ago

https://github.com/huggingface/text-generation-inference

Main features of TGI are quite awesome. It woud be nice to make it additional inference implementation.

fardeon commented 1 year ago

Thanks for introducing TGU, we'll definitely check it out!