langchain-openai>=0.1.17 adds logprobs parameter to gpt-4vision requests which leads to an error in AzureOpenAI

blaufink commented 3 months ago

Checked other resources

[X] I added a very descriptive title to this issue.
[X] I searched the LangChain documentation with the integrated search.
[X] I used the GitHub search to find a similar question and didn't find it.
[X] I am sure that this is a bug in LangChain rather than my code.
[X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

from langchain.schema import HumanMessage
from langchain_openai import AzureChatOpenAI

llm = AzureChatOpenAI(
        base_url="BASE_URL" + "/deployments/" + "gpt-4v",
        openai_api_version = "2024-02-01",
        api_key="API-KEY"
    )
message = HumanMessage(content="""{
            "role": "system",
            "content": "You are a helpful assistant and can help with identifying or making assumptions about content in images."
        },
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Describe this picture:"
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/e/e3/Plains_Zebra_Equus_quagga.jpg/800px-Plains_Zebra_Equus_quagga.jpg"
                    }
                }
            ]
        }""")
print(llm.invoke([message]))

Error Message and Stack Trace (if applicable)

This leads to the following error:

openai.BadRequestError: Error code: 400 - {'error': {'message': '1 validation error for Request\nbody -> logprobs\n extra fields not permitted (type=value_error.extra)', 'type': 'invalid_request_error', 'param': None, 'code': None}}

Description

The error only occurs when using langchain-openai>=0.1.17 and can be attributed to the following PR: https://github.com/langchain-ai/langchain/pull/23691 Here, the parameter logprobs is added to requests per default. However, AzureOpenAI takes issue with this parameter as stated here: https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/chatgpt?tabs=python-new&pivots=programming-language-chat-completions -> "If you set any of these parameters, you get an error."

(Using langchain-openai<=0.1.16 or even adding a # comment in front of the logprobs addition in the site-package file circumvents the issue)

System Info

langchain==0.2.11 langchain-core==0.2.25 langchain-mistralai==0.1.11 langchain-openai==0.1.19 langchain-text-splitters==0.2.2

ccurme commented 3 months ago

Hello, thanks for this. Could you clarify what models have this issue? I'm having trouble reproducing it.

A subsequent PR also updated the behavior so that parameters (such as logprobs) are not included if they're None: https://github.com/langchain-ai/langchain/pull/24280. Does that not correct the issue?

blaufink commented 3 months ago

Hi, thanks for the reply. In my case, only the GPT4-v model has this issue.

I was able to further pinpoint the issue and found out why the subsequent PR you mentioned didn't solve this problem. In the implementation which introduced this error, all parameters' default values were set to None except for logprobs (set to False): https://github.com/langchain-ai/langchain/pull/23691/files#diff-197e34e7a297cef90049a95d035073870fb484896bab67f800a95fe190ab02bfR328
This is also the reason why it is not filtered before the request is made.

ccurme commented 3 months ago

Closing as I believe this is resolved by https://github.com/langchain-ai/langchain/pull/25229 but will re-open if needed.

Thank you for the fix.

langchain-ai / langchain