Open krrishdholakia opened 11 months ago
cc: @gururise can you add some details on how you're trying to use litellm with haystack promptnode?
+1 to this. LiteLLM could be the backend for all text generation and embedding node because it's highly compatible with a LOT of providers
Hello @krrishdholakia, Is there a plan in place to address this feature request? Thanks!
hey @lambda-science @kjoth , what is needed on our end for this?
Hi @krrishdholakia , Just as in Langchain and Lamaindex, where integrations for LiteLLM allow us to access LLM providers either through an LLM Proxy Server or directly via LiteLLM integration, a similar functionality is needed for Haystack. In Haystack, they refer to this functionality as "Generators," offering multiple integrations for accessing LLM providers. You can find more information about this at the provided link.
LamaIndex-LiteLLM Integration: https://docs.llamaindex.ai/en/stable/examples/llm/litellm/ Langchian-LiteLLm Integration: https://python.langchain.com/docs/integrations/chat/litellm/ Haystack Intrgeations with LLM Providers: https://github.com/deepset-ai/haystack-core-integrations?tab=readme-ov-file Haystack Generators for calling LLM Providers: https://docs.haystack.deepset.ai/docs/generators
With a LiteLLM generator integration for Haystack, it becomes possible to utilize LiteLLM within the Haystack framework for making LLM calls. If additional information is needed, please feel free to ask.
This Feqture R has been already mentioned in this Haystack PR link and Haystack doesnt seem to integrate it right now it seems.
The Feature
user request
Motivation, pitch
user request
Twitter / LinkedIn details
No response