search

BerriAI / litellm

Python SDK, Proxy Server to call 100+ LLM APIs using the OpenAI format - [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, Replicate, Groq]
https://docs.litellm.ai/docs/
Other
11.64k stars 1.34k forks source link

[Bug-langfuse]: `Item exceeds size limit (size: 2308149), dropping item.` - langfuse #2992

Open Manouchehri opened 4 months ago

Manouchehri commented 4 months ago

What happened?

When using Gemini 1.5 Pro with huge prompts, they aren't logged to Langfuse.

Related to https://github.com/BerriAI/litellm/issues/2055.

Relevant log output

No response

Twitter / LinkedIn details

https://www.linkedin.com/in/davidmanouchehri/

Manouchehri commented 4 months ago

This was the response for context. :) 

{
  "id": "chatcmpl-e5a91875-5be9-4ee1-9168-79e69dd4e4e5",
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "logprobs": null,
      "message": {
        "content": "removed",
        "role": "assistant",
        "function_call": null,
        "tool_calls": null
      }
    }
  ],
  "created": 1712955115,
  "model": "gemini-1.5-pro-preview-0409",
  "object": "chat.completion",
  "system_fingerprint": null,
  "usage": {
    "completion_tokens": 650,
    "prompt_tokens": 616179,
    "total_tokens": 616829
  }
}
ishaan-jaff commented 4 months ago

+1 seeing this on embedding too

ishaan-jaff commented 4 months ago

spoke to the langfuse team - they will push a fix for this by end of this week

ishaan-jaff commented 4 months ago

closing since this is not litellm related

Manouchehri commented 4 months ago

Do you know which ticket was made on the Langfuse side? :)

ishaan-jaff commented 4 months ago

cc @marcklingen

marcklingen commented 3 months ago

@Manouchehri https://github.com/langfuse/langfuse/issues/1676

Manouchehri commented 2 months ago

Hmm, we should just truncate the input to 1MB on LiteLLM to avoid this:

https://github.com/langfuse/langfuse-python/blob/9d027f66eb4db431f46152982a5571c5e18c5241/langfuse/task_manager.py#L27