search

deepset-ai / haystack

:mag: LLM orchestration framework to build customizable, production-ready LLM applications. Connect components (models, vector DBs, file converters) to pipelines or agents that can interact with your data. With advanced retrieval methods, it's best suited for building RAG, question answering, semantic search or conversational agent chatbots.
https://haystack.deepset.ai
Apache License 2.0
14.59k stars 1.71k forks source link

Support gRPC interface of TEI #7522

Open mhillebrand opened 3 months ago

mhillebrand commented 3 months ago

The gRPC interface of HuggingFace's Text Embeddings Inference (TEI) library is much, much faster than its HTTP interface. It would be fantastic to see Haystack add support for it.

https://github.com/huggingface/text-embeddings-inference?tab=readme-ov-file#grpc

anakin87 commented 3 months ago

Nice idea!

I had a look and seems non-trivial because we are currently using the huggingface_hub library to send requests to TEI and this library does not seem to support gRPC.

Let's see if they will introduce this feature...