search

BerriAI / litellm

Call all LLM APIs using the OpenAI format. Use Bedrock, Azure, OpenAI, Cohere, Anthropic, Ollama, Sagemaker, HuggingFace, Replicate (100+ LLMs)
https://docs.litellm.ai/docs/
Other
10.16k stars 1.13k forks source link

[Bug]: `traceparent` causes `extra_headers` on config to be ignored/overwritten #4448

Closed Manouchehri closed 3 days ago

Manouchehri commented 3 days ago

What happened?

I believe b16b846711937a1145e8519693f23310cfb885ad introduced a problem where whenever traceparent is present, it completely overrides/replaces the extra_headers configs I have in my yaml.

Relevant log output

No response

Twitter / LinkedIn details

https://www.linkedin.com/in/davidmanouchehri/

ishaan-jaff commented 3 days ago

from looking at the code it does not look like it's override any other extra_header when traceparent is present

Manouchehri commented 3 days ago

if "extra_headers" not in data: kinda sounds like it's only checking the incoming request and not the config.

ishaan-jaff commented 3 days ago

it should not check the config that is intended

Manouchehri commented 3 days ago

But... now this is overriding my config unexpectedly.

ishaan-jaff commented 3 days ago

is this how you set it on your config ? 

model_list:
  - model_name: fake-openai-endpoint
    litellm_params:
      model: openai/fake
      api_key: fake-key
      api_base: https://exampleopenaiendpoint-production.up.railway.app/
      extra_headers: {"X-Custom-Header": "value"}
Manouchehri commented 3 days ago

Yes.

Manouchehri commented 3 days ago

Confirmed. I reverted b16b846711937a1145e8519693f23310cfb885ad and now my extra_headers is used from my config like expected.

traceparent is set automatically by Google Cloud Run, it's not something I want to override my intentional config.

ishaan-jaff commented 3 days ago

@Manouchehri do you use the otel callback ? perhaps we can only run this for users using OTEL through litellm

Manouchehri commented 3 days ago

I don't use it at the moment.

ishaan-jaff commented 3 days ago

fixed here @Manouchehri https://github.com/BerriAI/litellm/commit/413877d1c6427a9e9c7e4badc146b978337f92d8