[Question]: Return Source Nodes when using Multi Document ReAct Agents

Question Validation

[X] I have searched both the documentation and discord for an answer.

Question

I am using ReAct agents to query multiple documents. I setted up a QueryEngineTool for every Document based on

from llama_index.core.tools import QueryEngineTool, ToolMetadata

from llama_index.core import SummaryIndex
from llama_index.core.agent import ReActAgent

# Build agents dictionary
agents = {}

for d in files:
    file_name = d["file_name"]
    title = d["title"]
    vector_index = VectorStoreIndex.from_vector_store(vector_store=pipeline.vector_store)
    vector_query_engine = vector_index.as_query_engine(
        similarity_top_k=5,
        filters=MetadataFilters(
            filters=[
                ExactMatchFilter(key="file_name", value=f"{file_name}"),
            ]
        ),
    )

    docs_in_file = get_docs_for_file(file_name)
    summary_index = SummaryIndex(docs_in_file)
    summary_query_engine = summary_index.as_query_engine()

    # define tools
    query_engine_tools = [
        QueryEngineTool(
            query_engine=vector_query_engine,
            metadata=ToolMetadata(
                name="vector_tool",
                description=(
                    "Useful for questions related to specific aspects of the scientific paper"
                    f"with the title {title} and filename {file_name}"
                ),
            ),
        ),
        QueryEngineTool(
            query_engine=summary_query_engine,
            metadata=ToolMetadata(
                name="summary_tool",
                description=(
                    "Useful for any requests that require comparisions or a holistic summary"
                    f" of EVERYTHING about the scientific paper with the title {title} and filename {file_name}. For questions about"
                    " more specific sections, please use the vector_tool."
                ),
            ),
        ),
    ]

    # build agent
    agent = ReActAgent.from_tools(
        query_engine_tools,
        llm=llm,
        verbose=False,
        system_prompt=f"""\
        You are a specialized agent designed to answer queries about a scientific paper with the title {title} in file {file_name}.
        You must ALWAYS use at least one of the tools provided when answering a question; do NOT rely on prior knowledge.\
        """
    )

    agents[str(file_name)] = agent

And a top agent

all_tools = []
for d in files:
    file_name = d["file_name"]
    title = d["title"]
    tool_name = str(file_name).replace(".","_")
    wiki_summary = (
        f"This content contains pages of the scientific paper with the title {title} from file {file_name}. Use"
        f" this tool if you want to answer any questions about {title} from file {file_name}.\n"
    )
    doc_tool = QueryEngineTool(
        query_engine=agents[str(file_name)],
        metadata=ToolMetadata(
            name=f"tool_{tool_name}",
            description=wiki_summary,
        )
    )
    all_tools.append(doc_tool)

from llama_index.core import VectorStoreIndex
from llama_index.core.objects import ObjectIndex

obj_index = ObjectIndex.from_objects(
    all_tools,
    index_cls=VectorStoreIndex,
)

from llama_index.storage.chat_store.redis import RedisChatStore
from llama_index.core.memory import ChatMemoryBuffer

chat_store = RedisChatStore(redis_client=redis_client, ttl=30000)

chat_memory = ChatMemoryBuffer.from_defaults(
    token_limit=3000,
    chat_store=chat_store,
    chat_store_key="user1",
)

from llama_index.core.agent import ReActAgent
top_agent = ReActAgent.from_tools(tool_retriever=obj_index.as_retriever(similarity_top_k=5), 
                              memory=chat_memory, 
                              llm=llm, 
                              system_prompt=""" \
You are a chat assistant designed to answer questions about scientific papers.\
Please ALWAYS use the tools provided to answer a question.\\
""",
                              verbose=False)

When i use a single VectorStoreIndex.as_query_engine() instance and execute query() i get the source nodes returned. If i perform top_agent.chat() no Source Nodes are returned.

How can i return the source nodes when using the top_agent?

To address the issue of not receiving source nodes in responses from your top-level ReAct Agent, despite individual document-specific agents returning them as expected, you'll need to make adjustments to ensure that source nodes are aggregated and included in the top agent's responses. Here's a concise guide to achieve this:

Ensure Document Agents Return Source Nodes: First, verify that each document-specific agent is correctly configured to return source nodes with their responses.
Modify Top-Level Agent to Include Source Nodes: Enhance the top-level agent to aggregate and include source nodes from the document-specific agents in its responses. This may involve customizing the response structure to ensure source nodes are part of the output.
Customize Tool Retriever Logic: Ensure the tool retriever mechanism used by the top-level agent is set up to consider source nodes in its operation. This might require customizing the retrieval process to include source nodes in the returned objects or in post-processing steps.
Implement Callbacks or Post-Processing Steps: If applicable, use callbacks or post-processing steps to attach source nodes from document-specific agents to the top-level agent's responses. This could involve adjusting the logic in these steps to handle source nodes appropriately.
Review Agent Construction Logic: Finally, review the construction logic of your top-level agent to ensure it's configured to handle source nodes correctly. This might mean passing additional parameters or adjusting internal mechanisms during the agent setup.

By following these steps, you can ensure that your top-level ReAct Agent returns source nodes in its responses, similar to the individual document-specific agents. This approach requires careful adjustment of the agent's configuration and possibly customizing certain components to handle source nodes as part of the response object.

#### Details

**Sources** - [docs/docs/examples/agent/multi_document_agents.ipynb](https://github.com/run-llama/llama_index/blob/main/docs/docs/examples/agent/multi_document_agents.ipynb) - [docs/docs/examples/agent/multi_document_agents-v1.ipynb](https://github.com/run-llama/llama_index/blob/main/docs/docs/examples/agent/multi_document_agents-v1.ipynb) - [docs/docs/examples/objects/object_index.ipynb](https://github.com/run-llama/llama_index/blob/main/docs/docs/examples/objects/object_index.ipynb) - [llama-index-core/llama_index/core/objects/base.py](https://github.com/run-llama/llama_index/blob/main/llama-index-core/llama_index/core/objects/base.py) - [llama-index-core/llama_index/core/objects/__init__.py](https://github.com/run-llama/llama_index/blob/main/llama-index-core/llama_index/core/objects/__init__.py) - [llama-index-core/tests/objects/test_base.py](https://github.com/run-llama/llama_index/blob/main/llama-index-core/tests/objects/test_base.py) - [llama-index-core/llama_index/core/tools/retriever_tool.py](https://github.com/run-llama/llama_index/blob/main/llama-index-core/llama_index/core/tools/retriever_tool.py) - [llama-index-core/llama_index/core/postprocessor/__init__.py](https://github.com/run-llama/llama_index/blob/main/llama-index-core/llama_index/core/postprocessor/__init__.py) - [llama-index-core/llama_index/core/indices/postprocessor.py](https://github.com/run-llama/llama_index/blob/main/llama-index-core/llama_index/core/indices/postprocessor.py) - [llama-index-core/llama_index/core/postprocessor/node.py](https://github.com/run-llama/llama_index/blob/main/llama-index-core/llama_index/core/postprocessor/node.py) - [llama-index-core/llama_index/core/postprocessor/types.py](https://github.com/run-llama/llama_index/blob/main/llama-index-core/llama_index/core/postprocessor/types.py) - [llama-index-legacy/llama_index/legacy/postprocessor/node.py](https://github.com/run-llama/llama_index/blob/main/llama-index-legacy/llama_index/legacy/postprocessor/node.py) - [llama-index-core/llama_index/core/agent/react/base.py](https://github.com/run-llama/llama_index/blob/main/llama-index-core/llama_index/core/agent/react/base.py) - [docs/docs/module_guides/indexing/vector_store_index.md](https://github.com/run-llama/llama_index/blob/main/docs/docs/module_guides/indexing/vector_store_index.md) - [llama-index-core/llama_index/core/memory/chat_memory_buffer.py](https://github.com/run-llama/llama_index/blob/main/llama-index-core/llama_index/core/memory/chat_memory_buffer.py) - [llama-index-legacy/llama_index/legacy/memory/chat_memory_buffer.py](https://github.com/run-llama/llama_index/blob/main/llama-index-legacy/llama_index/legacy/memory/chat_memory_buffer.py) - [llama-index-core/llama_index/core/query_engine/retriever_query_engine.py](https://github.com/run-llama/llama_index/blob/main/llama-index-core/llama_index/core/query_engine/retriever_query_engine.py) - [llama-index-core/llama_index/core/tools/query_engine.py](https://github.com/run-llama/llama_index/blob/main/llama-index-core/llama_index/core/tools/query_engine.py) - [llama-index-core/llama_index/core/indices/list/base.py](https://github.com/run-llama/llama_index/blob/main/llama-index-core/llama_index/core/indices/list/base.py) - [llama-index-core/llama_index/core/indices/list/retrievers.py](https://github.com/run-llama/llama_index/blob/main/llama-index-core/llama_index/core/indices/list/retrievers.py)
**About Dosu** This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

run-llama / llama_index

[Question]: Return Source Nodes when using Multi Document ReAct Agents #13287

Question Validation

Question