[Question]: OpenAIAgent puts {"content": null} in message history before calling the tools

[JulianOestreich90]

Question Validation

• [X] I have searched both the documentation and discord for an answer.

Question

I am using the OpenAI Top Level Agent as in the Multi Documents Agent Example. I made the agent use a persistent Redis memory. After querying something that involves using a tool, it writes the following entries to Redis:

0
{ "type": "user", "content": "Hi my name is Julian." }
1
{ "type": "assistant", "content": "Hello Julian! How can I assist you today?" }
2
{ "type": "user", "content": "Tell me about 'ChartQA'." }
3
{ "type": "assistant", "content": null }
4
{ "type": "tool", "content": "ChartQA involves utilizing a feedforward neural network language model (NNLM) to rescore the n-best lists of a robust machine translation (MT) baseline. This method has proven to enhance translation quality, with researchers investigating ways to integrate source language information into the NNLM for improved rescoring performance. The approach has shown success in enhancing translation quality compared to baseline methods." }
5
{ "type": "assistant", "content": "ChartQA involves utilizing a feedforward neural network language model (NNLM) to rescore the n-best lists of a robust machine translation (MT) baseline. This method has proven to enhance translation quality, with researchers investigating ways to integrate source language information into the NNLM for improved rescoring performance. The approach has shown success in enhancing translation quality compared to baseline methods." }

So message 3 is written in the database and it contains a null value, which is not allowed. Further queries using the memory will give me then: Retrying llama_index.llms.openai.base.OpenAI._chat in 0.699491702621411 seconds as it raised BadRequestError: Error code: 400 - {'error': {'message': "Invalid value for 'content': expected a string, got null.", 'type': 'invalid_request_error', 'param': 'messages.[4].content', 'code': None}}.

I know that i could instantiate a CustomMemoryBuffer that checks for null-contents before uploading. But instead i would like to understand why my agent is returning { "type": "assistant", "content": null } before calling the tools?

Here is how i initialized the agents:

from llama_index.core.tools import QueryEngineTool, ToolMetadata
from llama_index.agent.openai import OpenAIAgent

from llama_index.core import SummaryIndex

# Build agents dictionary
agents = {}

for file_name in files:
    vector_index = VectorStoreIndex.from_vector_store(vector_store=pipeline.vector_store)
    vector_query_engine = vector_index.as_query_engine(
        similarity_top_k=10,
        filters=MetadataFilters(
            filters=[
                ExactMatchFilter(key="file_name", value=f"{file_name}"),
            ]
        ),
    )

    docs_in_file = get_docs_for_file(file_name)
    summary_index = SummaryIndex(docs_in_file)
    summary_query_engine = summary_index.as_query_engine()

    # define tools
    query_engine_tools = [
        QueryEngineTool(
            query_engine=vector_query_engine,
            metadata=ToolMetadata(
                name="vector_tool",
                description=(
                    "Useful for questions related to specific aspects of the scientific paper"
                    f" {file_name}."
                ),
            ),
        ),
        QueryEngineTool(
            query_engine=summary_query_engine,
            metadata=ToolMetadata(
                name="summary_tool",
                description=(
                    "Useful for any requests that require comparisions or a holistic summary"
                    f" of EVERYTHING about the scientific paper {file_name}. For questions about"
                    " more specific sections, please use the vector_tool."
                ),
            ),
        ),
    ]

    # build agent
    agent = OpenAIAgent.from_tools(
        query_engine_tools,
        verbose=True,
        system_prompt=f"""\
You are a specialized agent designed to answer queries about {file_name}.
You must ALWAYS use at least one of the tools provided when answering a question; do NOT rely on prior knowledge.\
""",
    )

    agents[str(file_name)] = agent

all_tools = []
lc_tools = []
for file_name in files:
    tool_name = str(file_name).replace(".","_")
    wiki_summary = (
        f"This content contains pages of the scientific paper {file_name}. Use"
        f" this tool if you want to answer any questions about {file_name}.\n"
    )
    doc_tool = QueryEngineTool(
        query_engine=agents[str(file_name)],
        metadata=ToolMetadata(
            name=f"tool_{tool_name}",
            description=wiki_summary,
        )
    )
    all_tools.append(doc_tool)

from llama_index.core import VectorStoreIndex
from llama_index.core.objects import ObjectIndex

obj_index = ObjectIndex.from_objects(
    all_tools,
    index_cls=VectorStoreIndex,
)

from llama_index.agent.openai import OpenAIAgent
from llama_index.storage.chat_store.redis import RedisChatStore
from llama_index.core.memory import ChatMemoryBuffer

chat_store = RedisChatStore(redis_url="redis://localhost:6379", ttl=30000)

chat_memory = ChatMemoryBuffer.from_defaults(
    token_limit=3000,
    chat_store=chat_store,
    chat_store_key="user1",
)

top_agent = OpenAIAgent.from_tools(
    tool_retriever=obj_index.as_retriever(similarity_top_k=10),
    memory=chat_memory,
    system_prompt=""" \
You are an agent designed to answer queries about a set of given scientific papers.
Please always use the tools provided to answer a question. Do not rely on prior knowledge.\

""",
    verbose=True,
)

[dosubot[bot]]

The issue you're encountering, where your agent returns { "type": "assistant", "content": null } before calling the tools, likely stems from how the tool's output is handled. Specifically, if a tool execution is successful but the tool returns a None value or an incorrectly formatted ToolOutput object, this scenario isn't explicitly managed, leading to the null content issue.

To address this, ensure that every tool returns a ToolOutput object with a non-null content field upon successful execution. Additionally, you can modify the call_tool_with_error_handling function to check for None or invalid outputs from the tool and handle these cases appropriately. For instance, you could raise an exception or set a default error message if the tool's output is not as expected.

Here's a code snippet to help you modify the call_tool_with_error_handling function:

def call_tool_with_error_handling(
    tool: BaseTool,
    input_dict: Dict,
    error_message: Optional[str] = None,
    raise_error: bool = False,
) -> ToolOutput:
    try {
        output = tool(**input_dict)
        if output.content is None:  # Check if the content is None
            raise ValueError("Tool returned None content")
        return output
    } catch (Exception e) {
        if raise_error:
            throw
        error_message = error_message or f"Error: {e!s}"
        return ToolOutput(
            content=error_message,
            tool_name=tool.metadata.name,
            raw_input={"kwargs": input_dict},
            raw_output=e,
        )
    }

This modification ensures that if a tool returns a ToolOutput object with content set to None, it's treated as an error, and an appropriate error message is generated. This should prevent the agent from returning invalid responses and mitigate the BadRequestError you're encountering.

Sources

• llama-index-integrations/agent/llama-index-agent-openai-legacy/llama_index/agent/openai_legacy/openai_agent.py

#### About Dosu

This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot.

[JulianOestreich90]

@dosu-bot And where should i call the call_tool_with_error_handling function within my code?

[dosubot[bot]]

To integrate the call_tool_with_error_handling function within your code, you should override the call_tool method in the OpenAIAgent class. This method is responsible for calling the tools, and by overriding it, you can ensure that every tool call is processed through your error handling function. Here's a simplified example of how to do it:

class OpenAIAgent(Agent):
    # ...

    def call_tool(self, tool: BaseTool, input_dict: Dict) -> ToolOutput:
        return call_tool_with_error_handling(tool, input_dict)

    # ...

This modification directs all tool calls to go through call_tool_with_error_handling, which checks for None content and handles errors appropriately. If your tools involve asynchronous operations, you should also apply a similar override to the acall_tool method for asynchronous handling.

If there are any specific aspects of your call_tool_with_error_handling function or other parts of your code that you're unsure about, please share those details for more targeted advice.

Sources

• llama-index-legacy/llama_index/legacy/agent/legacy/openai_agent.py
• llama-index-legacy/llama_index/legacy/tools/tool_spec/base.py
• llama-index-core/llama_index/core/tools/calling.py

#### About Dosu

This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot.

[JulianOestreich90]

@dosu-bot I fixed it by changing message.content to an empty string if its None, by impementing a custom add_message() method for the RedisVectorStore:

from llama_index.agent.openai import OpenAIAgent
from llama_index.storage.chat_store.redis import RedisChatStore
from llama_index.core.memory import ChatMemoryBuffer

class SafeRedisChatStore(RedisChatStore):
    def add_message(self, key, message):
        print(f"Storing message: {message.content}")       
        if message.content is None:
            message.content = ""
        super().add_message(key, message)

chat_store = SafeRedisChatStore(redis_url="redis://localhost:6379", ttl=30000)

chat_memory = ChatMemoryBuffer.from_defaults(
    token_limit=3000,
    chat_store=chat_store,
    chat_store_key="user1",
)

top_agent = OpenAIAgent.from_tools(
    tool_retriever=obj_index.as_retriever(similarity_top_k=10),
    memory=chat_memory,
    system_prompt=""" \
You are an agent designed to answer queries about a set of given scientific papers.
Please always use the tools provided to answer a question. Do not rely on prior knowledge. Do not issue a response with \{'content': null\}\

""",
    verbose=True,
)

Now i run into a new Error:

Retrying llama_index.llms.openai.base.OpenAI._chat in 0.9066396127181989 seconds as it raised BadRequestError: Error code: 400 - {'error': {'message': "Invalid parameter: messages with role 'tool' must be a response to a preceeding message with 'tool_calls'.", 'type': 'invalid_request_error', 'param': 'messages.[5].role', 'code': None}}.

How can i fix this?

[dosubot[bot]]

The error you're encountering, where messages with the role 'tool' must follow a 'tool_calls' message, is a specific requirement from the OpenAI API. This means that in your message flow, every 'tool' message must be directly preceded by a 'tool_calls' message to satisfy the API's constraints.

To address this, you can modify the add_message() method in your custom SafeRedisChatStore class to enforce this rule. Before adding a 'tool' message, check if the last message stored has the role 'tool_calls'. If not, you should either adjust the message flow to include a 'tool_calls' message before the 'tool' message or handle the error accordingly.

Here's a conceptual approach to modifying the add_message() method:

class SafeRedisChatStore(RedisChatStore):
    def add_message(self, key, message):
        if message.content is None:
            message.content = ""
        last_message = self.get_last_message(key)
        if message.role == 'tool' and (last_message is None or last_message.role != 'tool_calls'):
            raise ValueError("A 'tool' message must follow a 'tool_calls' message.")
        super().add_message(key, message)

This code snippet includes a check for the last message's role before adding a new 'tool' message. It's a simplified example, and you'll need to implement the get_last_message() method or an equivalent to retrieve the last message for the given key.

This solution ensures that your message flow adheres to the OpenAI API's requirements, preventing the BadRequestError you're experiencing.

Sources

• llama-index-legacy/llama_index/legacy/storage/chat_store/redis_chat_store.py
• llama-index-legacy/llama_index/legacy/core/llms/types.py

#### About Dosu

This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot.

[JulianOestreich90]

@dosu-bot There are no 'tool_calls' messages added to the chat_history, when using ChatMemoryBuffer.

[dosubot[bot]]

To resolve the issue where 'tool_calls' messages are not being added to the chat history when using ChatMemoryBuffer, it's crucial to ensure that your application correctly handles the sequence and structure of messages according to the OpenAI API's requirements. Specifically, a 'tool' message must directly follow a 'tool_calls' message. Here's a focused approach to address this:

1. Ensure Proper Message Structure: Verify that messages intended for tool calls (tool_calls) are correctly structured. This involves setting up messages with the appropriate fields such as id, type, function, etc., to comply with the OpenAI API's expectations.

2. Update Message Handling Logic: Your application's logic for processing, converting, and storing chat messages must recognize and correctly handle the tool_calls role. This may require adjustments in enums, validation logic, or storage schemas to accommodate tool_calls as a valid message role.

3. Implement Error Handling for Message Roles: Enhance your error handling to manage BadRequestError related to incorrect message role sequences. This should help in identifying and rectifying issues with message role handling promptly.

4. Testing and Validation: Extend your testing to cover scenarios that involve tool_calls, ensuring that your application can handle these messages both when sending to and receiving from the OpenAI API.

For practical examples and more detailed guidance, you might look into the test suites and class definitions within the llama-index repository, specifically focusing on how tool_calls are structured and handled. This includes looking at test cases for message conversion functions and the OpenAIToolCall class for encapsulating tool call information.

By following these steps and ensuring that your application adheres to the OpenAI API's message handling requirements, you should be able to mitigate the error related to 'tool' messages needing to follow 'tool_calls' messages in your chat history.

Sources

• [Bug]: Intermittent 400 - Invalid parameter Error for Messages with Role tool
• [Bug]: RedisChatStorage doesn't support OpenAI tool messages
• llama-index-integrations/llms/llama-index-llms-openai/tests/test_openai_utils.py
• llama-index-integrations/agent/llama-index-agent-openai/llama_index/agent/openai/step.py

#### About Dosu

This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot.