[Bug]: Intermittent 400 - Invalid parameter Error for Messages with Role tool

zurek11 commented 8 months ago

Bug Description

We are encountering intermittent 400 errors with the message "Invalid parameter: messages with role 'tool' must be a response to a preceding message with 'tool_calls'." This issue arises when interacting with our AI Bot Client, which leverages the LlamaIndex for querying and chat functionalities. The error does not always occur, but it presents a significant challenge when it does, particularly for straightforward requests like 'Tell me a joke'.

Version

0.9.44

Steps to Reproduce

Initialize the AI Bot Client with a specific configuration for the LlamaIndex, including storage context setup.

class AIBotClient:
    ai_bot_client = None

    def __init__(self):
        openai.api_key = settings.OPEN_AI_API_KEY

        self._llm = OpenAI(model_name=settings.OPEN_AI_MODEL, max_retries=5)  # gpt-4-turbo-preview
        self._service_context = ServiceContext.from_defaults(llm=self._llm)
        self._persist_storage = settings.BASE_DIR / 'apps/core/ai_bot/knowledge_index'
        self._storage_context = StorageContext.from_defaults(persist_dir=self._persist_storage)
        self._index = load_index_from_storage(self._storage_context)
        self._chat_engine = self._index.as_chat_engine(service_context=self._service_context)

Send a message to the bot via our FastAPI endpoint, which internally calls the chat method of the AI Bot Client.

    @staticmethod
    async def get_user_context(user_id: UUID) -> str:
        conversation_history = await Conversation.filter(user_id=user_id).order_by('created_at').all()
        conversation_messages = "\n".join([conv.message for conv in conversation_history])
        history_prompt = (
            f"This is your previous conversation history with the current user: {conversation_messages}\n "
            f"and this is his hew message he needs answer to:\n\n"
        )
        return history_prompt

    async def chat(self, user_id: UUID, message: str) -> str:
        context = await self.get_user_context(user_id)
        response = self._chat_engine.chat(context + message).response
        return response

The error occurs intermittently when the chat method processes the user's message and interacts with the underlying LlamaIndex services.

Relevant Logs/Tracbacks

{'error': {'message': "Invalid parameter: messages with role 'tool' must be a response to a preceding message with 'tool_calls'.", 'type': 'invalid_request_error', 'param': 'messages.[0].role', 'code': None}}

Full traceback from FastAPI when error occurs
---------------------------------------
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/venv/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 426, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/fastapi/applications.py", line 1106, in __call__
    await super().__call__(scope, receive, send)
  File "/venv/lib/python3.11/site-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/venv/lib/python3.11/site-packages/starlette/middleware/base.py", line 108, in __call__
    response = await self.dispatch_func(request, call_next)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/apps/api/middlewares/transaction.py", line 7, in db_transaction_wrapper
    response = await call_next(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/starlette/middleware/base.py", line 84, in call_next
    raise app_exc
  File "/venv/lib/python3.11/site-packages/starlette/middleware/base.py", line 70, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "/venv/lib/python3.11/site-packages/starlette/middleware/cors.py", line 83, in __call__
    await self.app(scope, receive, send)
  File "/venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/venv/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__
    raise e
  File "/venv/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__
    await self.app(scope, receive, send)
  File "/venv/lib/python3.11/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/venv/lib/python3.11/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/venv/lib/python3.11/site-packages/starlette/routing.py", line 66, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/fastapi/routing.py", line 274, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/apps/api/routers/ai_bot/conversation.py", line 27, in add_conversation
    bot_response = await ai_bot_client.chat(user_id=user.id, message=conversation_form.message)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/apps/core/clients/aibot.py", line 58, in chat
    response = self._chat_engine.chat(context + message).response
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/llama_index/callbacks/utils.py", line 41, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/llama_index/agent/runner/base.py", line 497, in chat
    chat_response = self._chat(
                    ^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/llama_index/agent/runner/base.py", line 442, in _chat
    cur_step_output = self._run_step(
                      ^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/llama_index/agent/runner/base.py", line 304, in _run_step
    cur_step_output = self.agent_worker.run_step(step, task, **kwargs)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/llama_index/callbacks/utils.py", line 41, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/llama_index/agent/openai/step.py", line 573, in run_step
    return self._run_step(
           ^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/llama_index/agent/openai/step.py", line 448, in _run_step
    agent_chat_response = self._get_agent_response(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/llama_index/agent/openai/step.py", line 322, in _get_agent_response
    chat_response: ChatResponse = self._llm.chat(**llm_chat_kwargs)
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/llama_index/llms/base.py", line 100, in wrapped_llm_chat
    f_return_val = f(_self, messages, **kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/llama_index/llms/openai.py", line 237, in chat
    return chat_fn(messages, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/llama_index/llms/openai.py", line 296, in _chat
    response = client.chat.completions.create(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/openai/_utils/_utils.py", line 303, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/openai/resources/chat/completions.py", line 598, in create
    return self._post(
           ^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/openai/_base_client.py", line 1086, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/openai/_base_client.py", line 846, in request
    return self._request(
           ^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/openai/_base_client.py", line 898, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError: Error code: 400 - {'error': {'message': "Invalid parameter: messages with role 'tool' must be a response to a preceeding message with 'tool_calls'.", 'type': 'invalid_request_error', 'param': 'messages.[0].role', 'code': None}}

dosubot[bot] commented 8 months ago

🤖

Based on the error message you're encountering, it seems like there's an issue with the sequence of messages being sent to the AI Bot Client. The error message suggests that a message with the role 'tool' is being sent without a preceding message with 'tool_calls'.

In the LlamaIndex framework, 'tool' and 'tool_calls' are used to handle function calls within the system. The 'tool' is an instance of a BaseTool class, which represents a function that can be called within the system. The 'tool_calls' are instances of the OpenAIToolCall class, which represent requests to call a function. Each 'tool_call' contains an 'id', a 'function' (which includes the function name and arguments), and possibly other information. The 'tool_calls' are processed in the _acall_function and _call_function methods, where each 'tool_call' is used to call the corresponding 'tool' and the output is stored.

The error could be occurring if there is a 'tool' message being sent without a corresponding 'tool_calls' message. This could be due to a bug in the code or due to the sequence of messages being sent to the AI Bot Client.

To resolve this issue, ensure that each 'tool' message is preceded by a 'tool_calls' message. This could involve checking the sequence of messages being sent to the AI Bot Client and debugging the _run_step and _arun_step methods to ensure they are correctly handling the tool calls and messages.

You can find more information about how 'tool' and 'tool_calls' are used in the LlamaIndex framework in the llama_index/agent/openai/step.py and llama_index/agent/legacy/openai_agent.py files.

I hope this helps! If you have any more questions or need further clarification, feel free to ask.

Sources

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

logan-markewich commented 8 months ago

Yea this is due to the memory buffer excluding messages, causing chat history to get into an invalid state for openai

needs to be fixed by adding some logic to the memory buffer to avoid doing this

zurek11 commented 8 months ago

I encountered the same issue on the OpenAI Community Forum: Link

This might be helpful.

tinnocente2023 commented 7 months ago

hi @zurek11, have you been able to work around this? I am running into the same issue

run-llama / llama_index