search

langchain-ai / langchain

🦜🔗 Build context-aware reasoning applications
https://python.langchain.com
MIT License
93.43k stars 15.04k forks source link

KeyError '*' in parse_result when using `with_structured_output` on ChatAnthropic #27260

Open srtab opened 3 days ago

srtab commented 3 days ago

Checked other resources

Example Code

from langchain_anthropic import ChatAnthropic
from pydantic import BaseModel, ConfigDict, Field
from langchain_core.messages import SystemMessage, HumanMessage

class RequestAssessmentResponse(BaseModel):
    """
    Respond to the reviewer's feedback with an assessment of the requested changes.
    """

    model_config = ConfigDict(title="request_assessment")

    request_for_changes: bool = Field(description="Set to True if the reviewer requested changes; otherwise, False.")
    justification: str = Field(description="Justify why you think it's a change request.")

model = ChatAnthropic(model="claude-3-5-sonnet-20240620").with_structured_output(RequestAssessmentResponse, method="json_schema")
model.invoke([SystemMessage("Your goal is to determine whether the comment is a direct request for changes or not"), HumanMessage("Change the name of the file foo.py.")])

Error Message and Stack Trace (if applicable)

KeyError                                  Traceback (most recent call last)
Cell In[1], line 17
     13     justification: str = Field(description="Justify why you think it's a change request.")
     16 model = ChatAnthropic(model="claude-3-5-sonnet-20240620").with_structured_output(RequestAssessmentResponse, method="json_schema")
---> 17 model.invoke([SystemMessage("Your goal is to determine whether the comment is a direct request for changes or not"), HumanMessage("Change the name of the file foo.py.")])

File ~/.venv/lib/python3.12/site-packages/langchain_core/runnables/base.py:3024, in RunnableSequence.invoke(self, input, config, **kwargs)
   3022             input = context.run(step.invoke, input, config, **kwargs)
   3023         else:
-> 3024             input = context.run(step.invoke, input, config)
   3025 # finish the root run
   3026 except BaseException as e:

File ~/.venv/lib/python3.12/site-packages/langchain_core/output_parsers/base.py:193, in BaseOutputParser.invoke(self, input, config, **kwargs)
    186 def invoke(
    187     self,
    188     input: Union[str, BaseMessage],
    189     config: Optional[RunnableConfig] = None,
    190     **kwargs: Any,
    191 ) -> T:
    192     if isinstance(input, BaseMessage):
--> 193         return self._call_with_config(
    194             lambda inner_input: self.parse_result(
    195                 [ChatGeneration(message=inner_input)]
    196             ),
    197             input,
    198             config,
    199             run_type="parser",
    200         )
    201     else:
    202         return self._call_with_config(
    203             lambda inner_input: self.parse_result([Generation(text=inner_input)]),
    204             input,
    205             config,
    206             run_type="parser",
    207         )

File ~/.venv/lib/python3.12/site-packages/langchain_core/runnables/base.py:1927, in Runnable._call_with_config(self, func, input, config, run_type, serialized, **kwargs)
   1923     context = copy_context()
   1924     context.run(_set_config_context, child_config)
   1925     output = cast(
   1926         Output,
-> 1927         context.run(
   1928             call_func_with_variable_args,  # type: ignore[arg-type]
   1929             func,  # type: ignore[arg-type]
   1930             input,  # type: ignore[arg-type]
   1931             config,
   1932             run_manager,
   1933             **kwargs,
   1934         ),
   1935     )
   1936 except BaseException as e:
   1937     run_manager.on_chain_error(e)

File ~/.venv/lib/python3.12/site-packages/langchain_core/runnables/config.py:396, in call_func_with_variable_args(func, input, config, run_manager, **kwargs)
    394 if run_manager is not None and accepts_run_manager(func):
    395     kwargs["run_manager"] = run_manager
--> 396 return func(input, **kwargs)

File ~/.venv/lib/python3.12/site-packages/langchain_core/output_parsers/base.py:194, in BaseOutputParser.invoke.<locals>.<lambda>(inner_input)
    186 def invoke(
    187     self,
    188     input: Union[str, BaseMessage],
    189     config: Optional[RunnableConfig] = None,
    190     **kwargs: Any,
    191 ) -> T:
    192     if isinstance(input, BaseMessage):
    193         return self._call_with_config(
--> 194             lambda inner_input: self.parse_result(
    195                 [ChatGeneration(message=inner_input)]
    196             ),
    197             input,
    198             config,
    199             run_type="parser",
    200         )
    201     else:
    202         return self._call_with_config(
    203             lambda inner_input: self.parse_result([Generation(text=inner_input)]),
    204             input,
    205             config,
    206             run_type="parser",
    207         )

File ~/.venv/lib/python3.12/site-packages/langchain_core/output_parsers/openai_tools.py:293, in PydanticToolsParser.parse_result(self, result, partial)
    288         msg = (
    289             f"Tool arguments must be specified as a dict, received: "
    290             f"{res['args']}"
    291         )
    292         raise ValueError(msg)
--> 293     pydantic_objects.append(name_dict[res["type"]](**res["args"]))
    294 except (ValidationError, ValueError) as e:
    295     if partial:

KeyError: 'request_assessment'

Description

When i use Pydantic models with model_config = ConfigDict(title="request_assessment") the exception KeyError: 'request_assessment' is raised when i use ChatAnthropic. With ChatOpenAI, no problems.

System Info

System Information
------------------
> OS:  Linux
> OS Version:  #45-Ubuntu SMP PREEMPT_DYNAMIC Fri Aug 30 12:02:04 UTC 2024
> Python Version:  3.12.7 (main, Oct  1 2024, 22:28:49) [GCC 12.2.0]

Package Information
-------------------
> langchain_core: 0.3.10
> langchain: 0.3.3
> langchain_community: 0.3.2
> langsmith: 0.1.133
> langchain_anthropic: 0.2.3
> langchain_chroma: 0.1.4
> langchain_openai: 0.2.2
> langchain_text_splitters: 0.3.0
> langgraph: 0.2.35

Optional packages not installed
-------------------------------
> langserve

Other Dependencies
------------------
> aiohttp: 3.10.5
> anthropic: 0.36.0
> async-timeout: Installed. No version info available.
> chromadb: 0.5.3
> dataclasses-json: 0.6.7
> defusedxml: 0.7.1
> fastapi: 0.112.2
> httpx: 0.27.2
> jsonpatch: 1.33
> langgraph-checkpoint: 2.0.0
> numpy: 1.26.4
> openai: 1.51.2
> orjson: 3.10.7
> packaging: 24.1
> pydantic: 2.9.2
> pydantic-settings: 2.5.2
> PyYAML: 6.0.2
> requests: 2.32.3
> requests-toolbelt: 1.0.0
> SQLAlchemy: 2.0.32
> tenacity: 8.5.0
> tiktoken: 0.7.0
> typing-extensions: 4.12.2
srtab commented 3 days ago

After digging a bit on the code, it seams the error is not specifically related with ChatAnthropic but with BaseChatModel and method with_structured_output. The schema passed to the method is a Pydantic model, which means PydanticToolsParser is used. Checking the parse_result from PydanticToolsParser, the name_dict = {tool.__name__: tool for tool in self.tools} is built using the tool.__name__ as key. When the tools are bind, the method convert_to_openai_tool is used to convert Pydantic model to a tool and that method uses title generated by model_json_schema and not __name__, leading the a mismatch on the tool names.

My suggestion is change PydanticToolsParser to build name_dict using the result of convert_to_openai_tool(tool)["function"]["name"] for each tool: name_dict = {convert_to_openai_tool(tool)["function"]["name"] tool for tool in self.tools}.

If that's ok, i can open a pull request with a fix.