When I use the tool in Agent, it returns OPEN AI 400 Bad Request.

reinershir commented 3 months ago

Checked other resources

[X] I added a very descriptive title to this issue.
[X] I searched the LangChain documentation with the integrated search.
[X] I used the GitHub search to find a similar question and didn't find it.
[X] I am sure that this is a bug in LangChain rather than my code.
[X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

from langchain_community.utilities import SerpAPIWrapper
        from langchain.agents import create_openai_tools_agent
        from langchain.agents import AgentExecutor,Tool

        chat = ChatOpenAI(model="gpt-3.5-turbo-1106",streaming=True)

        search = SerpAPIWrapper()
        #search = GoogleSearchAPIWrapper()

        tools = [Tool(
            name="google_search",
            description="Search Google for recent results.",
            func=search.run,
            return_direct=False
        )]

        prompt = ChatPromptTemplate.from_messages(
            [
                (
                    "system",
                    "You are a helpful assistant. You may not need to use tools for every query - the user may just want to chat!",
                ),
                MessagesPlaceholder(variable_name="messages"),
                MessagesPlaceholder(variable_name="agent_scratchpad"),
            ]
        )
        agent = create_openai_tools_agent(tools = tools,llm = chat,prompt=prompt)
        agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

        for chunk in agent_executor.stream({"messages": chat_history.messages}):
                for key in chunk:
                    if key not in output:
                        output[key] = chunk[key]
                    else:
                        output[key] += chunk[key]
                if "actions" in chunk:
                    for action in chunk["actions"]:
                        print(f"Calling Tool: `{action.tool}` with input `{action.tool_input}`")
                    continue
                if "steps" in chunk:
                    observation = chunk["steps"][-1].observation
                    for step in chunk["steps"]:
                        print(f"Tool Result: `{step.observation}`")
                    continue

                if "output" in chunk:
                     print(chunk["output"], end="", flush=True)
                     response_json =  json.dumps({"stat": "SUCCESS", "content": chunk["output"]})

Error Message and Stack Trace (if applicable)

{'error': {'message': "An assistant message with 'tool_calls' must be followed by tool messages responding to each 'tool_call_id'. The following tool_call_ids did not have response messages: call_APhVboGGQV2ZfLqDnqukNltV", 'type': 'invalid_request_error', 'param': 'messages.[4].role', 'code': None}}

Description

I want to integrate Google search into my chatbot and use streaming output. It returned the following error: openai.BadRequestError: Error code: 400.

I searched on both Google and Github but did not find any relevant information.

System Info

System Information OS: Windows OS Version: 10.0.19045 Python Version: 3.12.1 (tags/v3.12.1:2305ca5, Dec 7 2023, 22:03:25) [MSC v.1937 64 bit (AMD64)]

Package Information langchain_core: 0.1.32 langchain: 0.1.12 langchain_community: 0.0.28 langsmith: 0.1.27 langchain_openai: 0.0.8 langchain_text_splitters: 0.0.1 langchainhub: 0.1.15

reinershir commented 3 months ago

More infomation:

Debugging middleware caught exception in streamed response at a point where response headers were already sent.
Traceback (most recent call last):
  File "C:\Users\developer\AppData\Local\Programs\Python\Python312\Lib\site-packages\werkzeug\wsgi.py", line 256, in __next__
    return self._next()
           ^^^^^^^^^^^^
  File "C:\Users\developer\AppData\Local\Programs\Python\Python312\Lib\site-packages\werkzeug\wrappers\response.py", line 32, in _iter_encoded
    for item in iterable:
  File "d:\Software\Develop\Projects\openai-python\sse-chatbot.py", line 233, in eventStream
    for chunk in chat_stream(project_id,user_id,data_type,question,"chat",path,chat_history):
  File "C:\Users\developer\AppData\Local\Programs\Python\Python312\Lib\site-packages\langchain\agents\agent.py", line 1571, in stream      
    for step in iterator:
  File "C:\Users\developer\AppData\Local\Programs\Python\Python312\Lib\site-packages\langchain\agents\agent_iterator.py", line 174, in __iter__
    for chunk in self.agent_executor._iter_next_step(
  File "C:\Users\developer\AppData\Local\Programs\Python\Python312\Lib\site-packages\langchain\agents\agent.py", line 1166, in _iter_next_step
    output = self.agent.plan(
             ^^^^^^^^^^^^^^^^
  File "C:\Users\developer\AppData\Local\Programs\Python\Python312\Lib\site-packages\langchain\agents\agent.py", line 514, in plan
    for chunk in self.runnable.stream(inputs, config={"callbacks": callbacks}):
  File "C:\Users\developer\AppData\Local\Programs\Python\Python312\Lib\site-packages\langchain_core\runnables\base.py", line 2589, in stream
    yield from self.transform(iter([input]), config, **kwargs)
  File "C:\Users\developer\AppData\Local\Programs\Python\Python312\Lib\site-packages\langchain_core\runnables\base.py", line 2576, in transform
    yield from self._transform_stream_with_config(
  File "C:\Users\developer\AppData\Local\Programs\Python\Python312\Lib\site-packages\langchain_core\runnables\base.py", line 1656, in _transform_stream_with_config
    chunk: Output = context.run(next, iterator)  # type: ignore
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\developer\AppData\Local\Programs\Python\Python312\Lib\site-packages\langchain_core\runnables\base.py", line 2540, in _transform
    for output in final_pipeline:
  File "C:\Users\developer\AppData\Local\Programs\Python\Python312\Lib\site-packages\langchain_core\runnables\base.py", line 1180, in transform
    for chunk in input:
  File "C:\Users\developer\AppData\Local\Programs\Python\Python312\Lib\site-packages\langchain_core\runnables\base.py", line 4430, in transform
    yield from self.bound.transform(
  File "C:\Users\developer\AppData\Local\Programs\Python\Python312\Lib\site-packages\langchain_core\runnables\base.py", line 1197, in transform
    yield from self.stream(final, config, **kwargs)
  File "C:\Users\developer\AppData\Local\Programs\Python\Python312\Lib\site-packages\langchain_core\language_models\chat_models.py", line 258, in stream
    raise e
  File "C:\Users\developer\AppData\Local\Programs\Python\Python312\Lib\site-packages\langchain_core\language_models\chat_models.py", line 241, in stream
    for chunk in self._stream(
  File "C:\Users\developer\AppData\Local\Programs\Python\Python312\Lib\site-packages\langchain_openai\chat_models\base.py", line 419, in _stream
    for chunk in self.client.create(messages=message_dicts, **params):
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\developer\AppData\Local\Programs\Python\Python312\Lib\site-packages\openai\_utils\_utils.py", line 275, in wrapper        
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\developer\AppData\Local\Programs\Python\Python312\Lib\site-packages\openai\resources\chat\completions.py", line 667, in create
    return self._post(
           ^^^^^^^^^^^
  File "C:\Users\developer\AppData\Local\Programs\Python\Python312\Lib\site-packages\openai\_base_client.py", line 1208, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\developer\AppData\Local\Programs\Python\Python312\Lib\site-packages\openai\_base_client.py", line 897, in request
    return self._request(
           ^^^^^^^^^^^^^^
  File "C:\Users\developer\AppData\Local\Programs\Python\Python312\Lib\site-packages\openai\_base_client.py", line 988, in _request        
    raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError: Error code: 400 - {'error': {'message': "An assistant message with 'tool_calls' must be followed by tool messages responding to each 'tool_call_id'. The following tool_call_ids did not have response messages: call_HNrotfEtm3IbJ6IYcKA6yEL9", 'type': 'invalid_request_error', 'param': 'messages.[3].role', 'code': None}}

liugddx commented 3 months ago

Set prompt to prompt = hub.pull("hwchase17/openai-tools-agent")

liugddx commented 3 months ago

Refer to https://python.langchain.com/docs/use_cases/tool_use/agents/#create-prompt

reinershir commented 3 months ago

Set prompt to prompt = hub.pull("hwchase17/openai-tools-agent") Does not work

eyurtsev commented 3 months ago

@reinershir this is a modified script that's missing information. While this could be a bug in LangChain, it's also impossible to tell whether it's actually an error in user code.

There's some parameter here that's not defined

chat_history.messages

I suggest following on the existing implementations and then adapting to your use case: https://python.langchain.com/docs/modules/agents/agent_types/tool_calling/

eyurtsev commented 3 months ago

@reinershir if you're able to share a minimal reproducible example, we could take a look at whether this is a bug on LangChain side

liugddx commented 3 months ago

This code will cause problems.

I found that when calling the openai interface, its additional parameters are {'tools': [{'function': {'description': 'Search Google for recent results.', 'name': 'google_search', 'parameters' : {'properties': {'__arg1': {...}}, 'required': ['__arg1'], 'type': 'object'}}, 'type': 'function'}]}

import json
from typing import List

from langchain_community.utilities import SerpAPIWrapper
from langchain.agents import create_openai_tools_agent
from langchain.agents import AgentExecutor, Tool
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
import os

chat = ChatOpenAI(model="gpt-3.5-turbo-1106", streaming=True)
from langchain_core.chat_history import BaseChatMessageHistory
from langchain_core.pydantic_v1 import BaseModel, Field
from langchain_core.messages import BaseMessage, AIMessage

search = SerpAPIWrapper()

# search = GoogleSearchAPIWrapper()

tools = [Tool(
    name="google_search",
    description="Search Google for recent results.",
    func=search.run,
    return_direct=False
)]

prompt = ChatPromptTemplate.from_messages(
    [
        (
            "system",
            "You are a helpful assistant. You may not need to use tools for every query - the user may just want to chat!",
        ),
        MessagesPlaceholder(variable_name="messages"),
        MessagesPlaceholder(variable_name="agent_scratchpad"),
    ]
)

class InMemoryHistory(BaseChatMessageHistory, BaseModel):
    """In memory implementation of chat message history."""

    messages: List[BaseMessage] = Field(default_factory=list)

    def add_messages(self, messages: List[BaseMessage]) -> None:
        """Add a list of messages to the store"""
        self.messages.extend(messages)

    def clear(self) -> None:
        self.messages = []

chat_history = InMemoryHistory()
agent = create_openai_tools_agent(tools=tools, llm=chat, prompt=prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)
output = {}

print(agent_executor.invoke({"messages": chat_history.messages}))
for chunk in agent_executor.stream({"messages": chat_history.messages}):
    for key in chunk:
        if key not in output:
            output[key] = chunk[key]
        else:
            output[key] += chunk[key]
    if "actions" in chunk:
        for action in chunk["actions"]:
            print(f"Calling Tool: `{action.tool}` with input `{action.tool_input}`")
        continue
    if "steps" in chunk:
        observation = chunk["steps"][-1].observation
        for step in chunk["steps"]:
            print(f"Tool Result: `{step.observation}`")
        continue

    if "output" in chunk:
        print(chunk["output"], end="", flush=True)
        response_json = json.dumps({"stat": "SUCCESS", "content": chunk["output"]})

reinershir commented 3 months ago

@eyurtsev
I have read the official documentation many times, but it did not solve my problem. Here is the complete runnable code:

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain.memory import ChatMessageHistory
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings
from langchain.chains.combine_documents import create_stuff_documents_chain
from flask import Flask, Response, request
import tiktoken
import os,random
# import openai
import json

import logging
import sys

logging.basicConfig(
                    level    = logging.INFO,                                                               
                    format   = '%(asctime)s  %(filename)s : %(levelname)s  %(message)s',   
                    datefmt  = '%Y-%m-%d %A %H:%M:%S',                                  
#                    filename = 'gpt-index.log',                
#                    filemode = 'w',
                    stream=sys.stdout)
os.environ["OPENAI_API_KEY"] = 'xxxxxxxxxxxxxxxxxx'

os.environ["SERPAPI_API_KEY"] = 'xxxxxxxxxxxxxxxxxxxxx'

app = Flask(__name__)

@app.after_request
def after_request(response):
    response.headers.add('Access-Control-Allow-Origin', '*')
    response.headers.add('Access-Control-Allow-Methods', 'GET,PUT,POST,DELETE,OPTIONS')
    response.headers.add('Access-Control-Allow-Headers', 'Content-Type,Authorization')
    response.headers.add('Access-Control-Allow-Credentials', 'true')
    return response

agent_cache={}
connecte_session = {}
session_cache={}
chat_history_cache={}

def get_chat_agent(project_id,session_id,data_type,path):
      # from langchain_community.utilities import GoogleSearchAPIWrapper
      from langchain_community.utilities import SerpAPIWrapper
      from langchain.agents import create_openai_tools_agent
      from langchain.agents import AgentExecutor,Tool
      chat = ChatOpenAI(model="gpt-3.5-turbo-1106",streaming=True)
      search = SerpAPIWrapper()
        tools = [Tool(
            name="google_search",
            description="Search Google for recent results.",
            func=search.results,
            return_direct=False
        )]
        prompt = ChatPromptTemplate.from_messages(
            [
                (
                    "system",
                    "You are a helpful assistant. You may not need to use tools for every query - the user may just want to chat!",
                ),
                # MessagesPlaceholder(variable_name="chat_history"),
                # ("human", "{input}"),
                MessagesPlaceholder(variable_name="messages"),
                MessagesPlaceholder(variable_name="agent_scratchpad"),
            ]
        )
        agent = create_openai_tools_agent(tools = tools,llm = chat,prompt=prompt)
        agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)
        return agent_executor

def chat_stream(project_id,session_id,data_type,question,intention,path,chat_history):
        chat_agent = get_chat_agent(project_id,session_id,data_type,path)
        chat_history.add_user_message(question)
        return chat_agent.stream({"messages": chat_history.messages},{"configurable": {"session_id": "unused"}},)

import json
import time
import datetime
from flask_cors import CORS,cross_origin

@cross_origin
@app.route('/stream')
def stream():
    user_id = request.args.get('userId')  
    data_type = request.args.get('dataType')  
    question = request.args.get('question') 
    project_id = request.args.get('projectId')
    path = request.args.get('path').replace("\\","/")

    def eventStream():
        event_name="chat_response"
        output = {}
        answer = ""
        chat_history = ChatMessageHistory()
        try: 
            for chunk in chat_stream(project_id,user_id,data_type,question,"chat",path,chat_history):
                time.sleep(0.05)
                for key in chunk:
                    if key not in output:
                        output[key] = chunk[key]
                    else:
                        output[key] += chunk[key]
                if "actions" in chunk:
                    for action in chunk["actions"]:
                        print(f"Calling Tool: `{action.tool}` with input `{action.tool_input}`")
                    continue
                if "steps" in chunk:
                    observation = chunk["steps"][-1].observation
                    for step in chunk["steps"]:
                        print(f"Tool Result: `{step.observation}`")
                    continue
                if "output" in chunk:
                    print(chunk["output"], end="", flush=True)
                    response_json =  json.dumps({"stat": "SUCCESS", "content": chunk["output"]})

                yield f'id: {id}\nevent: {event_name}\ndata:{response_json}\n\n'

            time.sleep(0.1)
            if("answer" in chunk):
                answer = output["answer"]
            else:
                answer = output["output"]
            chat_history.add_ai_message(answer)
            str_done = f'id: {id}\nevent: {event_name}\ndata: {json.dumps({"stat": "SUCCESS", "content": "[DONE]"})}\n\n'
            yield str_done

        except ValueError as e:
            print(e)
            response_json = {"stat": "FAILED", "content": str(e)}
            str_out = f'id: {id}\nevent: {event_name}\ndata: {json.dumps(response_json)}\n\n'
            yield str_out

    return Response(eventStream(), mimetype="text/event-stream")

if __name__ == '__main__':
    app.run(debug=True,host="0.0.0.0", port=8960)

request HTTP:`http://localhost:8960/stream?userId=any&dataType=any&question={yourQuestion}&projectId=any&path=any

liugddx commented 3 months ago

This problem is caused by repeatedly calling the tool in a session. Refer to https://community.openai.com/t/formatting-assistant-messages-after-tool-function-calls-in-gpt-conversations/535360

liugddx commented 3 months ago

https://smith.langchain.com/o/da397c07-cee5-579d-95ea-9ba07e28aea2/projects/p/5a2d376a-3d7e-436a-a06b-1dbdf9ab158d?timeModel=%7B%22duration%22%3A%227d%22%7D&tab=0&peek=81f996ac-6a8b-4c35-9887-bacdc0ef95a0. This is my test case

liugddx commented 3 months ago

Maybe the agent doesn‘t support streaming yet.

reinershir commented 3 months ago

Maybe the agent doesn‘t support streaming yet.

@liugddx
Thank you very much, you are right, it does not support streaming. Is there any other way to achieve streaming output?

eyurtsev commented 2 months ago

The agent is expected to support streaming. Could you confirm that this tutorial does not work https://python.langchain.com/docs/modules/agents/how_to/streaming/ ?

The issue is that the wrong information seems to be sent to the openai api -- it's missing a tool message. Would be good to confirm whether it's happening due to an issue at LangChain vs. an issue with using modified code

reinershir commented 2 months ago

The agent is expected to support streaming. Could you confirm that this tutorial does not work https://python.langchain.com/docs/modules/agents/how_to/streaming/ ?

The issue is that the wrong information seems to be sent to the openai api -- it's missing a tool message. Would be good to confirm whether it's happening due to an issue at LangChain vs. an issue with using modified code

I have carefully read the tutorial, the code is correct, everything works fine when I stop using streaming.

langchain-ai / langchain