search

Canner / WrenAI

🚀 An open-source SQL AI (Text-to-SQL) Agent that empowers data, product teams to chat with their data. 🤘
https://getwren.ai/oss
GNU Affero General Public License v3.0
2.04k stars 211 forks source link

litellm support #780

Open cyyeh opened 3 weeks ago

cyyeh commented 3 weeks ago

Currently we use Haystack to provide various LLM providers; however, we found it's still too hard for the community to contribute to various LLMs. We plan to use litellm instead; it provides more functionalities such as fallback, and it provides openai api compatible interface for all llms. We think it's easier for the community's contribution

cyyeh commented 3 weeks ago

This task is scheduled to assign to the contributor Gomaa

cyyeh commented 3 weeks ago

@MGomaa435

implementation plan doc

  1. We plan to replace implementation of LLMProvider and EmbedderProvider using litellm. However, we think we can focus on replacing LLMProvider first.
  2. For the get_generator method in each LLMProvider, we can implement a higher-order function that accepts model params and returns a function. when we invoke the returned function, it executes the llm generation api, which mean acompletion in litellm
  3. For all llm api calls, we need to use async version
  4. We can finish openai, openai-compatible, and ollama first?
MGomaa435 commented 2 weeks ago

Hi Jimmy, my pleasure to support in this