search

BerriAI / litellm

Python SDK, Proxy Server to call 100+ LLM APIs using the OpenAI format - [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, Replicate, Groq]
https://docs.litellm.ai/docs/
Other
12.16k stars 1.4k forks source link

[Feature]: Add litellm to haystack promptnode #591

Open krrishdholakia opened 11 months ago

krrishdholakia commented 11 months ago

The Feature

user request

Motivation, pitch

user request

Twitter / LinkedIn details

No response

krrishdholakia commented 11 months ago

cc: @gururise can you add some details on how you're trying to use litellm with haystack promptnode?

lambda-science commented 9 months ago

+1 to this. LiteLLM could be the backend for all text generation and embedding node because it's highly compatible with a LOT of providers

kjoth commented 4 months ago

Hello @krrishdholakia, Is there a plan in place to address this feature request? Thanks!

krrishdholakia commented 4 months ago

hey @lambda-science @kjoth , what is needed on our end for this?

kjoth commented 4 months ago

Hi @krrishdholakia , Just as in Langchain and Lamaindex, where integrations for LiteLLM allow us to access LLM providers either through an LLM Proxy Server or directly via LiteLLM integration, a similar functionality is needed for Haystack. In Haystack, they refer to this functionality as "Generators," offering multiple integrations for accessing LLM providers. You can find more information about this at the provided link.

LamaIndex-LiteLLM Integration: https://docs.llamaindex.ai/en/stable/examples/llm/litellm/ Langchian-LiteLLm Integration: https://python.langchain.com/docs/integrations/chat/litellm/ Haystack Intrgeations with LLM Providers: https://github.com/deepset-ai/haystack-core-integrations?tab=readme-ov-file Haystack Generators for calling LLM Providers: https://docs.haystack.deepset.ai/docs/generators

With a LiteLLM generator integration for Haystack, it becomes possible to utilize LiteLLM within the Haystack framework for making LLM calls. If additional information is needed, please feel free to ask.

kjoth commented 4 months ago

This Feqture R has been already mentioned in this Haystack PR link and Haystack doesnt seem to integrate it right now it seems.

https://github.com/deepset-ai/haystack/pull/5718