search

BerriAI / litellm

Call all LLM APIs using the OpenAI format. Use Bedrock, Azure, OpenAI, Cohere, Anthropic, Ollama, Sagemaker, HuggingFace, Replicate (100+ LLMs)
https://docs.litellm.ai/docs/
Other
10.16k stars 1.13k forks source link

[Bug]: LiteLLM Proxy Traceparent header is not forwarded to backends #4419

Closed aloababa closed 4 days ago

aloababa commented 4 days ago

What happened?

It seems that the Traceparent header is not forwarded to backends (vllm) when proxying with LiteLLM. It would be nice if it can be forwarded to the backends so the backend traces can have the LiteLLM traces as parent.

Relevant log output

Here is a dump of the request send to vllm by LiteLLM:

POST /v1/chat/completions HTTP/1.1
Host: mimproxy:6060
Accept: application/json
Accept-Encoding: gzip, deflate
Authorization: Bearer -
Connection: keep-alive
Content-Length: 144
Content-Type: application/json
User-Agent: AsyncOpenAI/Python 1.27.0
X-Stainless-Arch: x64
X-Stainless-Async: async:asyncio
X-Stainless-Lang: python
X-Stainless-Os: Linux
X-Stainless-Package-Version: 1.27.0
X-Stainless-Runtime: CPython
X-Stainless-Runtime-Version: 3.11.8

Twitter / LinkedIn details

No response

ishaan-jaff commented 4 days ago

is this with OTEL @aloababa ? So when you pass a traceparent header in the request -> litellm should add the traceparent header in the response

aloababa commented 4 days ago

Yes this is with OTEL, it successfully parse the Traceheader if there is one in the request so i can link my trace from frontend to LiteLLM, but after this Traceparent header is not forwarded to the backend (vllm) so my trace stops at LiteLLM and i got a new trace for vllm that is not linked to it.

ishaan-jaff commented 4 days ago

got it -> so if a traceparent is in the request header -> we should forward this to vLLM ?

ishaan-jaff commented 4 days ago

fix here @aloababa https://github.com/BerriAI/litellm/pull/4423

ishaan-jaff commented 4 days ago

I can see it forwards the traceparent to the provider

Screenshot 2024-06-26 at 12 34 39 PM
ishaan-jaff commented 4 days ago

@aloababa any chance we can hop on a call ? I'd love to learn how how we can improve litellm for you.

I reached out to you on Linkedin if DMs work. Sharing a link to my cal for your convenience: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

aloababa commented 4 days ago

Thank you very much for your reactivity. It works well if i provide the Traceparent header in the request to LiteLLM. Is it also possible that you forward the Traceparent header even if the Traceparent header is not provided in the original request ? so you put the trace id generated by the LiteLLM otel integration in the Traceparent header.