search

BerriAI / litellm

Python SDK, Proxy Server to call 100+ LLM APIs using the OpenAI format - [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, Replicate, Groq]
https://docs.litellm.ai/docs/
Other
12.05k stars 1.39k forks source link

[Feature]: Support Assistant API #2842

Closed jeromeroussin closed 4 months ago

jeromeroussin commented 5 months ago

The Feature

The Assistant API comes with new endpoints that liteLLM does not currently support. Docs:

We'd like to be able to proxy calls to the assistant API with litellm

Motivation, pitch

The assistant API is a quick way to provide "chat-with-your-doc" and codeInterpreter capabilities to an existing chatbot.

Twitter / LinkedIn details

https://www.linkedin.com/in/jeromeroussin/

krrishdholakia commented 4 months ago

acknowledging this issue @jeromeroussin

hoping to have a v0 out for testing by end of week

taralika commented 4 months ago

This would certainly be awesome, specifically looking for support for:

slavakurilyak commented 4 months ago

@krrishdholakia Any updates on this?

OpenAI Assistants API abstract away the complexity in handling stateful operations required for LLM-based apps including persistent threads, messages, and files.

This API unlocks automatic RAG pipelines so developers don't need to develop and manage their own vector-store infrastructure

Plus unifying data and LLMs within a single API is an underrated idea which saves developers time and ultimately money

krrishdholakia commented 4 months ago

hey @slavakurilyak @jeromeroussin @taralika we're hoping to have a v0 out for feedback by end of week.

krrishdholakia commented 4 months ago

Hi @taralika @jeromeroussin @slavakurilyak PR is now live. Aiming for the sdk support to be live today.

Is this something someone can give me feedback on next week? Help would be appreciated.

Next steps - adding azure + proxy support

slavakurilyak commented 4 months ago

Excited to see proxy support https://github.com/VRSEN/agency-swarm/issues/112

RussellLuo commented 3 months ago

Hi @krrishdholakia, sorry for replying in a closed issue.

I noticed that the current implementation of Assistants API only allows one provider openai:

https://github.com/BerriAI/litellm/blob/fda3914f4c8eaaf2d7f3ea2e2f1f0a711e31fdab/litellm/assistants/main.py#L43

Does this mean that we can only use OpenAI's models and cannot use other custom models such as Claude3, Llama3 or even Azure OpenAI's models for now?

If that's the case, is there any plan to support Assistants API for various models - like what LiteLLM did for chat/completion APIs - in the future?

Thanks!

phact commented 3 months ago

I'll be adding a provider that supports other models once https://github.com/datastax/astra-assistants-api/issues/22 is complete

krrishdholakia commented 3 months ago

@RussellLuo how would you suggest we support litellm's completion calls within the assistants api framework? this seems pretty provider specific.

I think the next step would be adding the azure endpoints

RussellLuo commented 3 months ago

@krrishdholakia Indeed, supporting the Assistants API requires a complete backend implementation. Looking forward to the support for Azure endpoints!

@phact Looks awesome, thanks for the great work!