search

run-llama / llama_index

LlamaIndex is a data framework for your LLM applications
https://docs.llamaindex.ai
MIT License
36.78k stars 5.27k forks source link

[Bug]: OpenAIAgent as Top Agent with Tool Retriever doesn't work - Multi-Document Agent #15515

Open sreeharissh14 opened 2 months ago

sreeharissh14 commented 2 months ago

Bug Description

I have a set of agents that are linked to vector and summary tools. I am using Azure OpenAI GPT-4o as LLM with OpenAIAgent. The top agent is not able to use the correct agentic tool during follow up question.

For eg. I have a agent called tool_hsg_max attached to top agent. When I try to chat with question like "what is the source of fund for hsg max product ?", it is able to call the "tool_hsg_max" and fetch the correct result. However, when I ask a follow up question like "what is the entry age requirement for this product?", the logs show "Tool with name tool_hsg_max not found, please double check.". This only occurs when I don't mention the product name in the question.

Is this because the tool_retriever is not able to get the context of the question from chat history?

Version

0.10.45

Steps to Reproduce

Follow up question to the top agent causes this issue in OpenAIAgent when using Azure OpenAI GPT-4o.

Relevant Logs/Tracbacks

No response

dosubot[bot] commented 2 months ago

Hey @sreeharissh14! I'm here to assist you with this bug you've encountered. I'm a bot designed to help resolve issues, answer questions, and guide you toward becoming a contributor. Let's work together to tackle this issue!

The issue you're encountering with the OpenAIAgent and Tool Retriever when using Azure OpenAI GPT-4o is likely due to the agent not recognizing the need to use the tool without explicit context or instructions. This is a known limitation where the agent fails to retrieve the correct tool for follow-up questions if the product name is not mentioned explicitly.

In your case, when you ask a follow-up question like "what is the entry age requirement for this product?" without mentioning the product name, the agent does not have enough context to identify the correct tool (tool_hsg_max). This happens because the agent does not automatically infer the context from the chat history.

To address this, you can try the following approaches:

  1. Explicit Context in Follow-Up Questions: Ensure that follow-up questions include explicit context or product names. For example, instead of "what is the entry age requirement for this product?", you could ask "what is the entry age requirement for the hsg max product?".

  2. Enhance Context Recognition: Adjust the configurations of the Tool Retriever to improve context recognition. For instance, you can set the similarity_top_k parameter to retrieve more relevant tools based on the context.

  3. Use CondensePlusContextChatEngine: Implement the CondensePlusContextChatEngine to handle ambiguous or incomplete follow-up questions. This engine uses a condense prompt to transform follow-up questions into standalone questions by incorporating the chat history, ensuring the correct tool is retrieved.

Here is an example of how to set up the CondensePlusContextChatEngine:

from llama_index.core.chat_engine.condense_plus_context import CondensePlusContextChatEngine

chat_engine = CondensePlusContextChatEngine(
    retriever=my_retriever,
    llm=my_llm,
    memory=my_memory,
    context_prompt="Your custom context prompt",
    condense_prompt="Your custom condense prompt",
    system_prompt="Your system prompt",
    skip_condense=False,
    node_postprocessors=[my_postprocessor],
    callback_manager=my_callback_manager,
    verbose=True
)

By using these approaches, you can improve the agent's ability to recognize and utilize context in follow-up questions, ensuring the correct tool is retrieved [1][2][3].

To continue talking to Dosu, mention @dosu.

logan-markewich commented 2 months ago

Probably because the tool retriever isn't returning the tool the second time, but the agent can see it in the chat history, so tries to call it anyways

sreeharissh14 commented 2 months ago

Yes, I have increased the number of similar results to the maximum and solved the issue. But, would like to understand if there is a cleaner approach to solve the problem? For eg, is it possible to pass context to ObjectIndex? Or are there other utilities for handling such cases? I have a lot of Document Agents and converse with the agents in a chat style(follow up question, etc).

sreeharissh14 commented 2 months ago

Hi @logan-markewich, can you suggest any mechanism to overcome this limitation for ObjectIndex?

sreeharissh14 commented 2 months ago

Hi @dosu, how do I use CondensePlusContextChatEngine as tool retriever? Give me example code with ObjectIndex and sample tools.