langchain-ai / langchain

🦜🔗 Build context-aware reasoning applications
https://python.langchain.com
MIT License
94.96k stars 15.38k forks source link

Agent Executor Submits Tool Ouputs Twice (Or shows same behaviour) #27757

Open burakcbdn opened 2 weeks ago

burakcbdn commented 2 weeks ago

Checked other resources

Example Code

gpt4o = ChatOpenAI(
    model="gpt-4o",
    streaming=True,
    extra_body={
        "response_format": {
            "type": "json_schema",
            "json_schema": {
                "name": "response",
                "strict": True,
                "schema": responseSchema
            }
        }
    },
    temperature=0.6,)
currentModel = gpt4o
showImageTool = ShowImageTool()
tools = [
    showImageTool
]

agent = create_tool_calling_agent(
  llm=toolCallingModel,
  tools=tools,
  prompt=prompt
)
agentExecutor = AgentExecutor(
    agent=agent,
    tools=tools,
    return_intermediate_steps=True,
    verbose=True
)

Error Message and Stack Trace (if applicable)

No response

Description

I have a AgentExecutor with only 1 tool call, the returned info from tool is not important so I am only returning "tool_done" from the tool execution.

After the execution done, tool outputs get submitted to the model and new step output starts to come, but at some point, model restarts to produce new output (like tool is resubmitted)

Verbose output is as follows:

Invoking: `show_image` with `{'imageId': 'test-id'}`
responded: {"content":["<some_content_array>"]}

image_shown: test-id{"content":["some_content_array", "after_image_shown"]}
{"content":["<some_new_content>"]}

The model has structured output, so output with this type breaks the structure. (The new content does not comes with new chat message, it just continues to stream like that)

I tried same structure and function in openAI playground, it works correctly in there, whether it calls the tool at the start, middle or end.

System Info

System Information
------------------
> OS:  Darwin
> OS Version:  Darwin Kernel Version 24.0.0: Mon Aug 12 20:52:18 PDT 2024; root:xnu-11215.1.10~2/RELEASE_ARM64_T8122
> Python Version:  3.11.10 (main, Sep  7 2024, 01:03:31) [Clang 15.0.0 (clang-1500.3.9.4)]

Package Information
-------------------
> langchain_core: 0.3.10
> langchain: 0.3.3
> langchain_community: 0.3.2
> langsmith: 0.1.133
> langchain_astradb: 0.5.0
> langchain_cli: 0.0.31
> langchain_openai: 0.2.2
> langchain_text_splitters: 0.3.0
> langchain_unstructured: 0.1.5
> langserve: 0.3.0

Optional packages not installed
-------------------------------
> langgraph

Other Dependencies
------------------
> aiohttp: 3.10.9
> astrapy: 1.5.2
> async-timeout: Installed. No version info available.
> dataclasses-json: 0.6.7
> fastapi: 0.112.4
> gitpython: 3.1.43
> gritql: 0.1.5
> httpx: 0.27.2
> jsonpatch: 1.33
> langserve[all]: Installed. No version info available.
> numpy: 1.26.4
> openai: 1.51.2
> orjson: 3.10.7
> packaging: 24.1
> pydantic: 2.9.2
> pydantic-settings: 2.5.2
> PyYAML: 6.0.2
> requests: 2.31.0
> requests-toolbelt: 1.0.0
> SQLAlchemy: 2.0.35
> sse-starlette: 1.8.2
> tenacity: 8.5.0
> tiktoken: 0.8.0
> tomlkit: 0.12.5
> typer[all]: Installed. No version info available.
> typing-extensions: 4.12.2
> unstructured-client: 0.25.9
> unstructured[all-docs]: Installed. No version info available.
> uvicorn: 0.23.2
burakcbdn commented 5 days ago

issue still persists