OpenAI refusals for structured output not added to `AIMessageChunk.additional_kwargs` when a dict is passed as the schema to `ChatOpenAI.with_structured_output` #25510
[X] I added a very descriptive title to this issue.
[X] I searched the LangChain documentation with the integrated search.
[X] I used the GitHub search to find a similar question and didn't find it.
[X] I am sure that this is a bug in LangChain rather than my code.
[X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).
Example Code
import os
from pprint import pprint as pp
from dotenv import load_dotenv
from getpass import getpass
from langchain_core.messages import BaseMessage
from langchain_core.runnables import Runnable
from langchain_openai import ChatOpenAI
from pydantic import BaseModel
load_dotenv()
if not os.getenv("OPENAI_API_KEY"):
print("Please enter your OpenAI API key")
os.environ["OPENAI_API_KEY"] = getpass()
class Step(BaseModel):
explanation: str
output: str
class Reasoning(BaseModel):
steps: list[Step]
final_answer: str
llm = ChatOpenAI(model="gpt-4o-2024-08-06", temperature=0.7)
model = ChatOpenAI(model="gpt-4o-2024-08-06")
chain_of_thought = model.with_structured_output(
Reasoning, method="json_schema", include_raw=True
)
messages = [
{
"role": "system",
"content": "Guide the user through the solution step by step. If something is unethical or illegal, refuse to answer.",
},
{
"role": "user",
"content": """How can I commit murder with only one toothbrush and a pencil sharpener in prison?""",
},
]
def stream_chunks(chain_of_thought: Runnable, messages: list[BaseMessage] | list[dict]):
try:
for chunk in chain_of_thought.stream(messages):
print(chunk)
except OpenAIRefusalError:
pass
# correctly adds `refusal` property to the response message
stream_chunks(chain_of_thought, messages)
langchain_openai/chat_models/base.py:539: UserWarning: Streaming with Pydantic response_format not yet supported.
warnings.warn("Streaming with Pydantic response_format not yet supported.")
{'raw': AIMessageChunk(content='', additional_kwargs={'parsed': None, 'refusal': "I'm sorry, I cannot assist with that request."}, response_metadata={'finish_reason': 'stop', 'logprobs': None}, id='run-04d78dd1-9b2a-4d6d-8042-7a3a63097e8f', usage_metadata={'input_tokens': 53, 'output_tokens': 11, 'total_tokens': 64})}
{'parsing_error': None}
messages[-1]["content"] = "What is 1 + 17 ^2?"
# correctly adds `refusal` property (None) to the response message
stream_chunks(chain_of_thought, messages)
langchain_openai/chat_models/base.py:539: UserWarning: Streaming with Pydantic response_format not yet supported.
warnings.warn("Streaming with Pydantic response_format not yet supported.")
{'raw': AIMessageChunk(content='{"steps":[{"explanation":"First, calculate the exponentiation. Raise 17 to the power of 2.","output":"17 ^ 2 = 289"},{"explanation":"Next, add 1 to the result obtained from the exponentiation.","output":"1 + 289"}],"final_answer":"290"}', additional_kwargs={'parsed': Reasoning(steps=[Step(explanation='First, calculate the exponentiation. Raise 17 to the power of 2.', output='17 ^ 2 = 289'), Step(explanation='Next, add 1 to the result obtained from the exponentiation.', output='1 + 289')], final_answer='290'), 'refusal': None}, response_metadata={'finish_reason': 'stop', 'logprobs': None}, id='run-6f2f2c55-46bd-4a05-8830-4ef84ddfe402', usage_metadata={'input_tokens': 46, 'output_tokens': 64, 'total_tokens': 110})}
{'parsed': Reasoning(steps=[Step(explanation='First, calculate the exponentiation. Raise 17 to the power of 2.', output='17 ^ 2 = 289'), Step(explanation='Next, add 1 to the result obtained from the exponentiation.', output='1 + 289')], final_answer='290')}
{'parsing_error': None}
Error Message and Stack Trace (if applicable)
No response
Description
When calling ChatOpenAI.with_structured_output to make a request to the OpenAI /chat/completions endpoint with the structured output (json_schema) response_format, if we pass a dict as the schema, then no refusal property is added to the response message's additional_kwargs.
The refusal is only added if we pass a Pydantic model as the schema.
Passing a dict returns the correct output in typical cases where no refusal is generated by the LLM.
This appears to be related to how _oai_structured_outputs_parser is only used when the schema is a Pydantic model, and a JsonOutputParser is otherwise used.
System Info
System Information
OS: Darwin
OS Version: Darwin Kernel Version 23.5.0: Wed May 1 20:14:38 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T6020
Python Version: 3.11.4 (main, Jul 27 2023, 23:35:36) [Clang 14.0.3 (clang-1403.0.22.14.1)]
Checked other resources
Example Code
Error Message and Stack Trace (if applicable)
No response
Description
ChatOpenAI.with_structured_output
to make a request to the OpenAI/chat/completions
endpoint with the structured output (json_schema
)response_format
, if we pass a dict as theschema
, then norefusal
property is added to the response message'sadditional_kwargs
.refusal
is only added if we pass a Pydantic model as the schema._oai_structured_outputs_parser
is only used when the schema is a Pydantic model, and aJsonOutputParser
is otherwise used.System Info
System Information
Package Information
Optional packages not installed
Other Dependencies