search

BerriAI / litellm

Call all LLM APIs using the OpenAI format. Use Bedrock, Azure, OpenAI, Cohere, Anthropic, Ollama, Sagemaker, HuggingFace, Replicate (100+ LLMs)
https://docs.litellm.ai/docs/
Other
10.16k stars 1.13k forks source link

[Bug]: Allow 2,097,152 tokens with Gemini 1.5 Pro #4450

Closed Manouchehri closed 3 days ago

Manouchehri commented 3 days ago

What happened?

I think this is just for Google AI Studio atm, not sure about Vertex AI.

image

Relevant log output

No response

Twitter / LinkedIn details

https://www.linkedin.com/in/davidmanouchehri/

emerzon commented 3 days ago

Also true for Vertex

Manouchehri commented 3 days ago

Ah yep, I see it's 2M on Vertex AI now too!

image
krrishdholakia commented 3 days ago

@Manouchehri can you clarify - what's the bug? Your screenshot is that of google ai studio

Manouchehri commented 3 days ago

@krrishdholakia https://github.com/BerriAI/litellm/blob/76490690a15af8c3b53a03878d7f268d0bf225af/model_prices_and_context_window.json#L1468

https://github.com/BerriAI/litellm/blob/76490690a15af8c3b53a03878d7f268d0bf225af/model_prices_and_context_window.json#L1493

krrishdholakia commented 3 days ago

added - https://github.com/BerriAI/litellm/commit/8b1bd749ffec45b44f3b0f1a8c4f8bbf1370af7f