[Bug]: passing extra headers to AzureOpenAI doesn't work

owenffff commented 1 month ago

What happened?

I'm trying to use AzureOpenAI API with litellm, for this i need to pass two extra headers. the following code doesn't work and returns 404 error:

Code:

username, password, gif_host, api_key, id_broker_host = (
    env_vars["username"], 
    env_vars["password"], 
    env_vars["gif_host"], 
    env_vars["api_key"], 
    env_vars["id_broker_host"]
)

#token is set somewhere else

response = litellm.completion(
    model="gpt-4-32k",
    api_base=gif_host,
    api_version="2023-07-01-preview",
    api_key=api_key,
    extra_headers={
        "Ocp-Apim-Subscription-Key": api_key,
        "Authorization": token
    },
    messages=[{"role": "user", "content": "good morning"}]
)

Output:

"NotFoundError: litellm.NotFoundError: NotFoundError: OpenAIException - Error code: 404 - {'statusCode': 404, 'message': 'Resource not found'}"

However, when i was trying to pass these extra headers using langchain's AzureChatOpenAI, it works:

Code:

from langchain_openai import AzureChatOpenAI

llm = AzureChatOpenAI(
    azure_endpoint=gif_host,
    api_version="2023-07-01-preview",
    api_key=api_key,
    default_headers={
        "Ocp-Apim-Subscription-Key": api_key,
        "Authorization": token
    },
    model_name="gpt-4-32k"
)

llm.invoke("tell me a joke")

Output:

AIMessage(content="Why don't scientists trust atoms?\n\nBecause they make up everything!", response_metadata={'token_usage': {'completion_tokens': 13, 'prompt_tokens': 11, 'total_tokens': 24}, 'model_name': 'gpt-4-32k', 'system_fingerprint': None, 'finish_reason': 'stop', 'logprobs': None, 'content_filter_results': {}}, id='run-0d01ef95-904b-416f-ba20-2e53faa39467-0', usage_metadata={'input_tokens': 11, 'output_tokens': 13, 'total_tokens': 24})

All the env variables are set correctly. Am I missing anything here?

Relevant log output

No response

Twitter / LinkedIn details

No response

Manouchehri commented 1 month ago

Having both Authorization and api_key confuses me a bit. Could you try dropping the Authorization header?

owenffff commented 1 month ago

Having both Authorization and api_key confuses me a bit. Could you try dropping the Authorization header?

cuz in original openai AzureOpenAI, api_key is a required field though not evaluated in this case. i tried dropping either Ocp-Apim-Subscription-Key or the Authorization header, both are not working..

krrishdholakia commented 1 month ago

can you share the raw request being made by litellm to azure @owenffff

litellm.set_verbose=True

owenffff commented 1 month ago

Request to litellm: litellm.completion(model='gpt-4-32k', api_base='{redacted}', api_version='2023-07-01-preview', api_key='{redacted}', extra_headers={'Ocp-Apim-Subscription-Key': '{redacted}', 'Authorization': 'Bearer eyJ0eXAiOiJKV1QiLCJraWQiOiJOaUppMUZZNDFYWnNIV2N1RDZoVXVFQyt1SjA9IiwiYWxnIjoiUlMyNTYifQ.eyJhdF9oYXNoIjoib3dGZ09zTFgyU0MweXNBSjVtWXFBQSIsImNvdW50cnkiOiJTRyIsInN1YiI6Im9saWFuZzAwNSIsImF1ZGl0VHJhY2tpbmdJZCI6IjcyYTM0MTAxLWNiYzktNGY3Mi04ZTQ4LTFjZGJkNzZmZjgwMy0xNTUwMDkyMzUiLCJpc3MiOiJodHRwczovL2xvZ2luLnB3Yy5jb206NDQzL29wZW5hbS9vYXV0aDIiLCJ0b2tlbk5hbWUiOiJpZF90b2tlbiIsImdpdmVuX25hbWUiOiJPd2VuIEwiLCJwcmVmZXJyZWRNYWlsIjoib3dlbi5sLmxpYW5nQHB3Yy5jb20iLCJMT1MiOiJUYXgiLCJ1aWQiOiJvbGlhbmcwMDUiLCJhdWQiOiJ1cm46eGRwOmlkYnJva2VyOnVzOnByZCIsInVwbiI6Im93ZW4ubC5saWFuZ0BzZy5wd2MuY29tIiwiYXpwIjoidXJuOnhkcDppZGJyb2tlcjp1czpwcmQiLCJhdXRoX3RpbWUiOjE3MjE3ODUwMjUsIm5hbWUiOiJPd2VuIEwgTGlhbmciLCJyZWFsbSI6Ii9wd2MiLCJleHAiOjE3MjE3ODg2MjUsInRva2VuVHlwZSI6IkpXVFRva2VuIiwiZmFtaWx5X25hbWUiOiJMaWFuZyIsImlhdCI6MTcyMTc4NTAyNSwiZW1haWwiOiJvd2VuLmwubGlhbmdAc2cucHdjLmNvbSJ9.KzNLC0vTgpDAsR5Kbd-HhBozWcYI2tL081MGH9E5GOMji2JsJG77z4HdRcCtT3qm4coYoWHwXR8YMfUtf3jjZjS0Y2_3E6JP-RiNr-wru9uYBlxOc8CP8ZlZgZ1dEc8kwND1YXZqIAxLAhpIPxBCDjtKSt97OAPU2Wf9vgvmDGMbF-zQ3vBnnUKw1QKHAwSKoyp05AqomcQ1uFOO7IrgOyiMAXn57AKW6d5X98eLHGCzWW_iSCFrGXvwT0rcnKk83GlSO41m8SDdk5mHB5W9idAxs_cbXOfN5mqr1VGj2yHN5WjkNhUb8AbGi4e4WKygnvvhzrp3CE1j30Vv0TYYfpkQ_TzJVxAzcSMtdvZV3UFKC3lRrbskyV9W4oVBM8RbbnVKn63Qa0qGvuWCsSnsrKy3KYIFEwans1DLrys0j29xMtaIup0_wrT3N80lknWTZTFZy2bfMVBcbCgnur6Va1BffSYTCgpI51XzlX-SWbHduv5wAQoMVkVK17y0GcNISXr3GDtO694C2Bis6hoBP2e9n-5shuceKvFC5dmfNLT7vluxIExPJy5E_Sr1F7yq4e8wf5N3jDWiU4FjRKpLfgwIA-U45EWJ0jtsvjHdEnhdtGsZ5facydAOTVjT5SUT6Ycx5yGWD9cPzsRaPRmpVtiDjvhzDMw3QVGlJXzh6Dc'}, messages=[{'role': 'user', 'content': 'good morning'}])

SYNC kwargs[caching]: False; litellm.cache: None; kwargs.get('cache')['no-cache']: False Final returned optional params: {'extra_headers': {'Ocp-Apim-Subscription-Key': '{redacted}', 'Authorization': 'Bearer eyJ0eXAiOiJKV1QiLCJraWQiOiJOaUppMUZZNDFYWnNIV2N1RDZoVXVFQyt1SjA9IiwiYWxnIjoiUlMyNTYifQ.eyJhdF9oYXNoIjoib3dGZ09zTFgyU0MweXNBSjVtWXFBQSIsImNvdW50cnkiOiJTRyIsInN1YiI6Im9saWFuZzAwNSIsImF1ZGl0VHJhY2tpbmdJZCI6IjcyYTM0MTAxLWNiYzktNGY3Mi04ZTQ4LTFjZGJkNzZmZjgwMy0xNTUwMDkyMzUiLCJpc3MiOiJodHRwczovL2xvZ2luLnB3Yy5jb206NDQzL29wZW5hbS9vYXV0aDIiLCJ0b2tlbk5hbWUiOiJpZF90b2tlbiIsImdpdmVuX25hbWUiOiJPd2VuIEwiLCJwcmVmZXJyZWRNYWlsIjoib3dlbi5sLmxpYW5nQHB3Yy5jb20iLCJMT1MiOiJUYXgiLCJ1aWQiOiJvbGlhbmcwMDUiLCJhdWQiOiJ1cm46eGRwOmlkYnJva2VyOnVzOnByZCIsInVwbiI6Im93ZW4ubC5saWFuZ0BzZy5wd2MuY29tIiwiYXpwIjoidXJuOnhkcDppZGJyb2tlcjp1czpwcmQiLCJhdXRoX3RpbWUiOjE3MjE3ODUwMjUsIm5hbWUiOiJPd2VuIEwgTGlhbmciLCJyZWFsbSI6Ii9wd2MiLCJleHAiOjE3MjE3ODg2MjUsInRva2VuVHlwZSI6IkpXVFRva2VuIiwiZmFtaWx5X25hbWUiOiJMaWFuZyIsImlhdCI6MTcyMTc4NTAyNSwiZW1haWwiOiJvd2VuLmwubGlhbmdAc2cucHdjLmNvbSJ9.KzNLC0vTgpDAsR5Kbd-HhBozWcYI2tL081MGH9E5GOMji2JsJG77z4HdRcCtT3qm4coYoWHwXR8YMfUtf3jjZjS0Y2_3E6JP-RiNr-wru9uYBlxOc8CP8ZlZgZ1dEc8kwND1YXZqIAxLAhpIPxBCDjtKSt97OAPU2Wf9vgvmDGMbF-zQ3vBnnUKw1QKHAwSKoyp05AqomcQ1uFOO7IrgOyiMAXn57AKW6d5X98eLHGCzWW_iSCFrGXvwT0rcnKk83GlSO41m8SDdk5mHB5W9idAxs_cbXOfN5mqr1VGj2yHN5WjkNhUb8AbGi4e4WKygnvvhzrp3CE1j30Vv0TYYfpkQ_TzJVxAzcSMtdvZV3UFKC3lRrbskyV9W4oVBM8RbbnVKn63Qa0qGvuWCsSnsrKy3KYIFEwans1DLrys0j29xMtaIup0_wrT3N80lknWTZTFZy2bfMVBcbCgnur6Va1BffSYTCgpI51XzlX-SWbHduv5wAQoMVkVK17y0GcNISXr3GDtO694C2Bis6hoBP2e9n-5shuceKvFC5dmfNLT7vluxIExPJy5E_Sr1F7yq4e8wf5N3jDWiU4FjRKpLfgwIA-U45EWJ0jtsvjHdEnhdtGsZ5facydAOTVjT5SUT6Ycx5yGWD9cPzsRaPRmpVtiDjvhzDMw3QVGlJXzh6Dc'}, 'extra_body': {}}

POST Request Sent from LiteLLM: curl -X POST \ {api_base_redacted} \ -H 'Ocp-Apim-Subscription-Key: *' -H 'Authorization: Bearer eyJ0eXAiOiJKV1QiLCJraWQiOiJOaUppMUZZNDFYWnNIV2N1RDZoVXVFQyt1SjA9IiwiYWxnIjoiUlMyNTYifQ.eyJhdF9oYXNoIjoib3dGZ09zTFgyU0MweXNBSjVtWXFBQSIsImNvdW50cnkiOiJTRyIsInN1YiI6Im9saWFuZzAwNSIsImF1ZGl0VHJhY2tpbmdJZCI6IjcyYTM0MTAxLWNiYzktNGY3Mi04ZTQ4LTFjZGJkNzZmZjgwMy0xNTUwMDkyMzUiLCJpc3MiOiJodHRwczovL2xvZ2luLnB3Yy5jb206NDQzL29wZW5hbS9vYXV0aDIiLCJ0b2tlbk5hbWUiOiJpZF90b2tlbiIsImdpdmVuX25hbWUiOiJPd2VuIEwiLCJwcmVmZXJyZWRNYWlsIjoib3dlbi5sLmxpYW5nQHB3Yy5jb20iLCJMT1MiOiJUYXgiLCJ1aWQiOiJvbGlhbmcwMDUiLCJhdWQiOiJ1cm46eGRwOmlkYnJva2VyOnVzOnByZCIsInVwbiI6Im93ZW4ubC5saWFuZ0BzZy5wd2MuY29tIiwiYXpwIjoidXJuOnhkcDppZGJyb2tlcjp1czpwcmQiLCJhdXRoX3RpbWUiOjE3MjE3ODUwMjUsIm5hbWUiOiJPd2VuIEwgTGlhbmciLCJyZWFsbSI6Ii9wd2MiLCJleHAiOjE3MjE3ODg2MjUsInRva2VuVHlwZSI6IkpXVFRva2VuIiwiZmFtaWx5X25hbWUiOiJMaWFuZyIsImlhdCI6MTcyMTc4NTAyNSwiZW1haWwiOiJvd2VuLmwubGlhbmdAc2cucHdjLmNvbSJ9.KzNLC0vTgpDAsR5Kbd-HhBozWcYI2tL081MGH9E5GOMji2JsJG77z4HdRcCtT3qm4coYoWHwXR8YMfUtf3jjZjS0Y2_3E6JP-RiNr-wru9uYBlxOc8CP8ZlZgZ1dEc8kwND1YXZqIAxLAhpIPxBCDjtKSt97OAPU2Wf9vgvmDGMbF-zQ3vBnnUKw1QKHAwSKoyp05AqomcQ1uFOO7IrgOyiMAXn57AKW6d5X98eLHGCzWW_iSCFrGXvwT0rcnKk83GlSO41m8SDdk5mHB5W9idAxs_cbXOfN5mqr1VGj2yHN5WjkNhUb8AbGi4e4WKygnvvhzrp3CE1j30Vv0TYYfpkQ_TzJVxAzcSMtdvZV3UFKC3lRrbskyV9W4oVBM8RbbnVKn63Qa0qGvuWCsSnsrKy3KYIFEwans1DLrys0j29xMtaIup0_wrT3N80lknWTZTFZy2bfMVBcbCgnur6Va1BffSYTCgpI51XzlX-SWbHduv5wAQoMVkVK17y0GcNISXr3GDtO694C2Bis6hoBP2e9n-5shuceKvFC5dmfNLT7vluxIExPJy5E_Sr1F7yq4e8wf5N3jDWiU4FjRKpLfgwIA-U45EWJ0jtsvjHdEnhdtGsZ5facydAOTVjT5SU****' \ -d '{'model': 'gpt-4-32k', 'messages': [{'role': 'user', 'content': 'good morning'}], 'extra_headers': {'Ocp-Apim-Subscription-Key': '{redacted}', 'Authorization': 'Bearer eyJ0eXAiOiJKV1QiLCJraWQiOiJOaUppMUZZNDFYWnNIV2N1RDZoVXVFQyt1SjA9IiwiYWxnIjoiUlMyNTYifQ.eyJhdF9oYXNoIjoib3dGZ09zTFgyU0MweXNBSjVtWXFBQSIsImNvdW50cnkiOiJTRyIsInN1YiI6Im9saWFuZzAwNSIsImF1ZGl0VHJhY2tpbmdJZCI6IjcyYTM0MTAxLWNiYzktNGY3Mi04ZTQ4LTFjZGJkNzZmZjgwMy0xNTUwMDkyMzUiLCJpc3MiOiJodHRwczovL2xvZ2luLnB3Yy5jb206NDQzL29wZW5hbS9vYXV0aDIiLCJ0b2tlbk5hbWUiOiJpZF90b2tlbiIsImdpdmVuX25hbWUiOiJPd2VuIEwiLCJwcmVmZXJyZWRNYWlsIjoib3dlbi5sLmxpYW5nQHB3Yy5jb20iLCJMT1MiOiJUYXgiLCJ1aWQiOiJvbGlhbmcwMDUiLCJhdWQiOiJ1cm46eGRwOmlkYnJva2VyOnVzOnByZCIsInVwbiI6Im93ZW4ubC5saWFuZ0BzZy5wd2MuY29tIiwiYXpwIjoidXJuOnhkcDppZGJyb2tlcjp1czpwcmQiLCJhdXRoX3RpbWUiOjE3MjE3ODUwMjUsIm5hbWUiOiJPd2VuIEwgTGlhbmciLCJyZWFsbSI6Ii9wd2MiLCJleHAiOjE3MjE3ODg2MjUsInRva2VuVHlwZSI6IkpXVFRva2VuIiwiZmFtaWx5X25hbWUiOiJMaWFuZyIsImlhdCI6MTcyMTc4NTAyNSwiZW1haWwiOiJvd2VuLmwubGlhbmdAc2cucHdjLmNvbSJ9.KzNLC0vTgpDAsR5Kbd-HhBozWcYI2tL081MGH9E5GOMji2JsJG77z4HdRcCtT3qm4coYoWHwXR8YMfUtf3jjZjS0Y2_3E6JP-RiNr-wru9uYBlxOc8CP8ZlZgZ1dEc8kwND1YXZqIAxLAhpIPxBCDjtKSt97OAPU2Wf9vgvmDGMbF-zQ3vBnnUKw1QKHAwSKoyp05AqomcQ1uFOO7IrgOyiMAXn57AKW6d5X98eLHGCzWW_iSCFrGXvwT0rcnKk83GlSO41m8SDdk5mHB5W9idAxs_cbXOfN5mqr1VGj2yHN5WjkNhUb8AbGi4e4WKygnvvhzrp3CE1j30Vv0TYYfpkQ_TzJVxAzcSMtdvZV3UFKC3lRrbskyV9W4oVBM8RbbnVKn63Qa0qGvuWCsSnsrKy3KYIFEwans1DLrys0j29xMtaIup0_wrT3N80lknWTZTFZy2bfMVBcbCgnur6Va1BffSYTCgpI51XzlX-SWbHduv5wAQoMVkVK17y0GcNISXr3GDtO694C2Bis6hoBP2e9n-5shuceKvFC5dmfNLT7vluxIExPJy5E_Sr1F7yq4e8wf5N3jDWiU4FjRKpLfgwIA-U45EWJ0jtsvjHdEnhdtGsZ5facydAOTVjT5SUT6Ycx5yGWD9cPzsRaPRmpVtiDjvhzDMw3QVGlJXzh6Dc'}, 'extra_body': {}}'

openai.py: Received openai error - Error code: 404 - {'statusCode': 404, 'message': 'Resource not found'} RAW RESPONSE: Error code: 404 - {'statusCode': 404, 'message': 'Resource not found'}

can you share the raw request being made by litellm to azure @owenffff

litellm.set_verbose=True

@krrishdholakia Thanks for helping! This is the full output i get, sry that i need to redact some field but overall the request seems fine right?

krrishdholakia commented 1 month ago

@owenffff is this redacted or actually missing

-H 'Ocp-Apim-Subscription-Key: '

i can see us making the request correctly, and passing extra headers as a param to chat.completion.create

owenffff commented 1 month ago

@owenffff is this redacted or actually missing

-H 'Ocp-Apim-Subscription-Key: '

i can see us making the request correctly, and passing extra headers as a param to chat.completion.create

for this specific field it shows "*****", so i guess it got redacted automatically

krrishdholakia commented 1 month ago

In my unit testing i can see it being set as expected

krrishdholakia commented 1 month ago

a difference i can see is one is setting extra_headers and the other is setting default_headers

krrishdholakia commented 1 month ago

@owenffff can you change

    "Authorization": token

to

extra_headers={"api-key": token}

and see if that fixes the issue

krrishdholakia commented 1 month ago

Langchain seems to ignore the correct key when i pass it as default_headers @owenffff

from langchain_openai import AzureChatOpenAI

llm = AzureChatOpenAI(
    azure_endpoint="https://openai-gpt-4-test-v-1.openai.azure.com",
    api_version="2023-07-01-preview",
    api_key="my-bad-key",
    default_headers={
        "Authorization": "Bearer good-key",
        "api-key": "good-key"
    },
    model_name="gpt-4-32k"
)

llm.invoke("tell me a joke")

could the error be something in the auth itself?

owenffff commented 1 month ago

@owenffff can you change
    "Authorization": token
to

extra_headers={"api-key": token}

and see if that fixes the issue

this did not fix the issue, as api key and bearer token are both required for Azure APIM.

owenffff commented 1 month ago

Langchain seems to ignore the correct key when i pass it as default_headers @owenffff Screenshot 2024-07-24 at 9 46 57 AM

from langchain_openai import AzureChatOpenAI

llm = AzureChatOpenAI(
    azure_endpoint="https://openai-gpt-4-test-v-1.openai.azure.com",
    api_version="2023-07-01-preview",
    api_key="my-bad-key",
    default_headers={
        "Authorization": "Bearer good-key",
        "api-key": "good-key"
    },
    model_name="gpt-4-32k"
)

llm.invoke("tell me a joke")

could the error be something in the auth itself?

For Langchain, it works similar to the original openai AzureOpenAI.

if we switch to the following code, the llm can still be initiated:

from langchain_openai import AzureChatOpenAI

llm = AzureChatOpenAI(
    azure_endpoint=gif_host,
    api_version="2024-06-01",
    api_key="a string required but not passed",
    default_headers={
        "Ocp-Apim-Subscription-Key": api_key,
        "Authorization": token
    },
    model_name="gpt-4-32k"
)

llm.invoke("tell me a joke")

Output:

AIMessage(content="Why don't scientists trust atoms?\n\nBecause they make up everything!", response_metadata={'token_usage': {'completion_tokens': 13, 'prompt_tokens': 11, 'total_tokens': 24}, 'model_name': 'gpt-4-32k', 'system_fingerprint': None, 'finish_reason': 'stop', 'logprobs': None, 'content_filter_results': {}}, id='run-f924257d-50e5-45d4-a9aa-a4691d75c2a4-0', usage_metadata={'input_tokens': 11, 'output_tokens': 13, 'total_tokens': 24})

So I guess my problem is more like how litellm handles the Azure APIM. I looked at the litellm azure source code but it's not that intuitive to me..

krrishdholakia commented 1 month ago

Hey @owenffff, i can see us correctly passing the extra headers for the azure request - can you help me understand what we're missing/doing incorrectly here?

Here's my colab with this code, for testing: https://colab.research.google.com/drive/1_kbxhbmX1tvbxFQ2K0Zrl5HbspVFQ2bf?usp=sharing

Also included this test in our ci/cd - https://github.com/BerriAI/litellm/commit/f35af3bf1c631f878b9be8fc207882383af5cf83

from httpx import Client
from unittest.mock import MagicMock, patch
from openai import AzureOpenAI
import litellm 
from litellm import completion
import os 

http_client = Client()

messages = [{"role": "user", "content": "Hello world"}]
with patch.object(http_client, "send", new=MagicMock()) as mock_client:
    litellm.client_session = http_client
    try:
        response = completion(
            model="azure/chatgpt-v-2",
            messages=messages,
            api_base="https://openai-gpt-4-test-v-1.openai.azure.com",
            api_version="2023-07-01-preview",
            api_key="my-azure-api-key",
            extra_headers={
                "Authorization": "my-bad-key",
                "Ocp-Apim-Subscription-Key": "hello-world-testing",
            },
        )
        print(response)
    except Exception as e:
        print(e)

    mock_client.assert_called()

    print(f"mock_client.call_args: {mock_client.call_args}")
    request = mock_client.call_args[0][0]
    print(request.method)  # This will print 'POST'
    print(request.url)  # This will print the full URL
    print(request.headers)  # This will print the full URL
    auth_header = request.headers.get("Authorization")
    apim_key = request.headers.get("Ocp-Apim-Subscription-Key")
    print(auth_header)
    assert auth_header == "my-bad-key"
    assert apim_key == "hello-world-testing"

BerriAI / litellm