Langfuse Integration ignore Embedding Output

[hburrichter]

Title

Langfuse Integration ignore Embedding Output

Relevant issues

Fixes #4225

Type

🐛 Bug Fix

Changes

1. When handling a litellm.EmbeddingResponse, set the Langfuse output to Null to avoid Langfuse counting the embedding vector as the observation/generation output.

2. Also I fixed one Langfuse test function create_async_task in which the Azure api version parameter was missing. Without the api version, the test fails:

E           Failed: An exception occurred - litellm.APIError: AzureException APIError - litellm.APIConnectionError: list index out of range
E           Traceback (most recent call last):
E             File "/Users/hannes/Repositories/litellm/litellm/main.py", line 834, in completion
E               optional_params = get_optional_params(
E                                 ^^^^^^^^^^^^^^^^^^^^
E             File "/Users/hannes/Repositories/litellm/litellm/utils.py", line 3149, in get_optional_params
E               optional_params = litellm.AzureOpenAIConfig().map_openai_params(
E                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E             File "/Users/hannes/Repositories/litellm/litellm/llms/azure.py", line 169, in map_openai_params
E               api_version_month = api_version_times[1]
E                                   ~~~~~~~~~~~~~~~~~^^^
E           IndexError: list index out of range
E           
E           During handling of the above exception, another exception occurred:
E           
E           Traceback (most recent call last):
E             File "/Users/hannes/Repositories/litellm/litellm/utils.py", line 6943, in exception_type
E               message=f"{exception_provider} APIConnectionError - {message}",
E                          ^^^^^^^^^^^^^^^^^^
E           UnboundLocalError: cannot access local variable 'exception_provider' where it is not associated with a value

test_alangfuse.py:146: Failed

[REQUIRED] Testing

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
litellm	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jun 16, 2024 2:17pm

[hburrichter]

@ishaan-jaff, thanks for the reply!

I want to make this change because the current output token count is inaccurate. As shown in the image I added to the linked issue, the Langfuse UI displays 28,410 output tokens. However, the actual number of generated tokens is only 10 + 278 = 288. Langfuse incorrectly includes the embedding output, a long floating-point vector, in the generated token count.

This makes it difficult to accurately track the number of tokens used in a trace, which is an important metric.

Additionally, I do not see any value in displaying the floating-point vector with all its values in the UI.

[krrishdholakia]

LGTM!