OpenAI Chat isn't forward compatible with OpenAI API

adubovik commented 1 month ago

Checked other resources

[X] I added a very descriptive title to this issue.
[X] I searched the LangChain documentation with the integrated search.
[X] I used the GitHub search to find a similar question and didn't find it.
[X] I am sure that this is a bug in LangChain rather than my code.
[X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

requirements.txt:

langchain==0.3.0
langchain-core==0.3.1
langchain-openai==0.2.0
openai==1.46.0
httpx==0.27.2

test.py:

from langchain_openai import ChatOpenAI
from langchain_openai.chat_models.base import _convert_dict_to_message, _convert_message_to_dict
from langchain_core.messages import HumanMessage, BaseMessage
import httpx
from pydantic import SecretStr

class MockClient(httpx.Client):
    def send(self, request, **kwargs):
        # !!! No extra_request field
        print(f"Request: {request.content.decode()}")

        message = {
            "role": "assistant",
            "content": "answer",
            "extra_response": "extra_response",
        }

        # !!! No extra_response field
        print(f"Response message dict: {_convert_dict_to_message(message)}")

        return httpx.Response(
            request=request,
            status_code=200,
            headers={"Content-Type": "application/json"},
            json={"choices": [{"index": 0, "message": message}]},
        )

llm = ChatOpenAI(api_key=SecretStr("-"), http_client=MockClient())

request_message = HumanMessage(content="question", extra_request="extra_request")

# !!! No extra_request field
print(f"Request message dict: {_convert_message_to_dict(request_message)}")

output = llm.generate(messages=[[request_message]])

response_message: BaseMessage = output.generations[0][0].message

# !!! No extra_response field
print(f"Response message: {response_message}")

> python test.py
Request message dict: {'content': 'question', 'role': 'user'}
Request: {"messages": [{"content": "question", "role": "user"}], "model": "gpt-3.5-turbo", "n": 1, "stream": false, "temperature": 0.7}
Response message dict: content='answer' additional_kwargs={} response_metadata={}
Response message: content='answer' additional_kwargs={'refusal': None} response_metadata={'token_usage': None, 'model_name': None, 'system_fingerprint': None, 'finish_reason': None, 'logprobs': None} id='run-7ba969eb-8a92-4c6d-9fc6-6a4a2f6d4bf6-0'

Error Message and Stack Trace (if applicable)

No response

Description

The core of the problem is that OpenAI adapter ignores additional fields in request messages and response messages. It happens in convert* family of methods where only a certain subset of fields from the input type is being converted.

There are two implications of this:

The forward compatibility of langchain_openai library with future versions of OpenAI API is undermined. Let's suppose OpenAI introduces a new field to the response assistant message, e.g. one which reflects thought process in the latest GPT-4 o1. The langchain_openai users will have to wait for the library to pick up with the changed in the API and then migrate their apps to the new version of the library. Same goes about the request messages. Curiously, the forward compatibility is supported for the top-level request parameters, which could be provided via extra_body parameter in ChatOpenAI.
Any custom extensions of the OpenAI API are not possible, since the library cut all of these extensions down.

Note that the openai library itself is designed to be forward compatible. The additional fields undeclared in the request and response schemas are passed through unperturbed. The request types are defined via TypedDict and the response one as pydantic BaseModel with extra fields allowed.

It would be great to achieve the same in langchain_openai as well.

System Info

aiohappyeyeballs==2.4.0
aiohttp==3.10.5
aiosignal==1.3.1
annotated-types==0.7.0
anyio==4.4.0
attrs==24.2.0
certifi==2024.8.30
charset-normalizer==3.3.2
distro==1.9.0
frozenlist==1.4.1
h11==0.14.0
httpcore==1.0.5
httpx==0.27.2
idna==3.10
jiter==0.5.0
jsonpatch==1.33
jsonpointer==3.0.0
langchain==0.3.0
langchain-core==0.3.1
langchain-openai==0.2.0
langchain-text-splitters==0.3.0
langsmith==0.1.121
multidict==6.1.0
numpy==1.26.4
openai==1.46.0
orjson==3.10.7
packaging==24.1
pydantic==2.9.2
pydantic_core==2.23.4
PyYAML==6.0.2
regex==2024.9.11
requests==2.32.3
sniffio==1.3.1
SQLAlchemy==2.0.35
tenacity==8.5.0
tiktoken==0.7.0
tqdm==4.66.5
typing_extensions==4.12.2
urllib3==2.2.3
yarl==1.11.1

mmmelkadri commented 5 days ago

Hello, I've tested your code and confirmed that certain fields are being ignored when converting between a dictionary and a message. I agree with you that this undermines forwards compatibility with future versions of OpenAI's API.

I'm a member of a group at University of Toronto and we're going to take a look at this issue and hopefully have a working pull request soon.

mmmelkadri commented 3 days ago

I've created a pull request with a fix for this issue.

While working on this issue, our group was wondering whether the new feature could accidentally break backwards compatibility in certain use cases (e.g. converting a dict where they want the extra fields to be ignored).

If anyone thinks this pull request might break their older code, our group would like to discuss this and see if there's another solution.

A group member also mentioned the possibility that allowing users to include extra fields may be against LangChain's goal, since it allows users to invoke messages that always result in errors. I haven't been using LangChain for long, so I'm not sure if that might be a consideration.

langchain-ai / langchain