search

BerriAI / litellm

Python SDK, Proxy Server (LLM Gateway) to call 100+ LLM APIs in OpenAI format - [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, Replicate, Groq]
https://docs.litellm.ai/docs/
Other
12.93k stars 1.51k forks source link

[Bug]: Llama Index Deepinfra bug #462

Closed krrishdholakia closed 11 months ago

krrishdholakia commented 1 year ago

What happened?

Using llama index with litellm. When trying to call deepinfra, it throws an error.

Relevant log output

No response

Twitter / LinkedIn details

No response

krrishdholakia commented 1 year ago

https://github.com/shauryr/S2QA/blob/litellm/notebooks/litellm.ipynb

Screenshot 2023-09-27 at 9 10 26 AM
krrishdholakia commented 1 year ago

cc: @shauryr

ishaan-jaff commented 1 year ago

im taking a look at this

I was able to fix this locally

ishaan-jaff commented 1 year ago

here's my deep infra call

Screenshot 2023-09-28 at 1 15 42 PM

TLDR: this is because our llama index implementation was trying to make text_completion requests for models like deep_infra

ishaan-jaff commented 1 year ago

@shauryr do you use llama index with streaming ?

shauryr commented 1 year ago

Yes, I use streaming

On Thu, Sep 28, 2023 at 3:17 PM Ishaan Jaff @.***> wrote:

@shauryr https://github.com/shauryr do you use llama index with streaming ?

— Reply to this email directly, view it on GitHub https://github.com/BerriAI/litellm/issues/462#issuecomment-1739948713, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADAFLTBYK7EADTEWOY6TF3DX4XLNTANCNFSM6AAAAAA5JTQY7Y . You are receiving this because you were mentioned.Message ID: @.***>

ishaan-jaff commented 1 year ago

I have a PR here it fixes regular calls for deep infra : https://github.com/jerryjliu/llama_index/pull/7885