Open anamariamv opened 11 months ago
Good day
Are there any options available in xef to use HuggingFace Transformers instead of the OpenAI API? In the context of the conversation()
block.
As far as I can tell, it's internally configured to use OpenAI. Is it possible to use Hugging Face models instead? Is that work in progress?
We're eager to use xef in production, but the high latency with the OpenAI requests is quite an inconvenience ... that's why we're curious about alternative options that xef has to offer
Hi @ff137!
We're working on adding more flexibility around the OpenAI API backend, allowing other servers with a compatible API to be used for particular requests. All this effort is initially planned for the Xef server, and its functionality can be shown in the following diagram:
sequenceDiagram
conversation()->>+OpenAIClient: SDK Call
OpenAIClient->>+XefServer: (OpenAI host = Xef Server)
XefServer->>OpenAIRest: Request
OpenAIRest-->>XefServer: Response
XefServer-->>-OpenAIClient:
OpenAIClient-->>-conversation():
This approach will also allow Xef users to implement custom mappings for servers with an API different from OpenAI.
This functionality is underway, and one of the features was added in this PR https://github.com/xebia-functional/xef/pull/513. Example 1
shows how to use huggingface/text-embeddings-inference for embeddings given a specific model.
Can we mark as completed, @anamariamv ?
Goals:
See what options we have to integrate transformers capabilities in xef.ai. Python bindings? Maybe creating our own native JVM version? Understand what training and fine-tuning models mean in the context of xef.ai.