[Bug]: ValueError: Tool with name functions not found, some queries will result in this error, while others won't.

Bug Description

I have modified according to this page, adding unique names to the metadata.

But the error still happend.

I have modified all the EnginTool's ToolMetadata

I also print the metadata for the retrieval step, ensuring that the names in the metadata are included.

I follow the these two Example, got same error.

Multi-Document Agents (V1) Multi-Document Agents

Version

0.10.38

Steps to Reproduce

response = agent.query("my_query")

Relevant Logs/Tracbacks

No response

To resolve the ValueError: Tool with name functions not found, ensure that the tool names are correctly defined and added to the list of tools. Here are the steps to address this issue:

Ensure Tool Names are Correctly Defined and Added

Define and Add Tools Correctly: Make sure that the tools are correctly defined and added to the list. For example:

from llama_index.core.tools import FunctionTool, QueryEngineTool, ToolMetadata

# Define FunctionTools
query_engine_tools = [
   FunctionTool(name="tool_1", metadata_tool=some_tool, description="Fetches specific aspects"),
   FunctionTool(name="tool_2", metadata_tool=another_tool, description="Provides summary")
]

# Define QueryEngineTools
all_tools = []
for file_name in file_names:
   comment = f"Processing {file_name}"
   tool = QueryEngineTool(ToolMetadata(name=file_name, index=idx), direct=False)
   all_tools.append(tool)

Pass the Tools List Correctly: Ensure that the tools list, including all defined tools, is passed to the relevant part of the system.
```
tools = query_engine_tools + all_tools
```

Modify the `get_function_by_name` Function

Ensure that the get_function_by_name function is correctly implemented to check for the existence of the tool name in the provided tools list.

def get_function_by_name(tools: List[BaseTool], name: str) -> BaseTool:
    """Get function by name."""
    name_to_tool = {tool.metadata.name: tool for tool in tools}
    if name not in name_to_tool:
        raise ValueError(f"Tool with name {name} not found")
    return name_to_tool[name]

Example Usage

Here's an example of how you might use the get_function_by_name function in your code:

# Assuming tools are defined and added to the list as shown above
tools = query_engine_tools + all_tools

# Correct tool name
tool_name = 'tool_1'

try:
    tool = get_function_by_name(tools, tool_name)
    print(f"Tool found: {tool}")
except ValueError as e:
    print(e)

Ensure Correct Imports

Make sure that the tools you are using are correctly imported and available:

from llama_index.core.tools import (
    BaseTool,
    FunctionTool,
    QueryEngineTool,
    RetrieverTool,
    ToolMetadata,
    ToolOutput,
)

Verify Tool Names

Ensure that the tool names you are using in your queries match the names defined in the ToolMetadata.

Additional Resources

Refer to the following documentation pages for more details on multi-document agents and tool usage:

By ensuring the correct definition, addition, and usage of tools, you can avoid the ValueError and ensure that the tool names are correctly recognized. If the issue persists, double-check the tool names and ensure they match exactly as defined in the ToolMetadata.

#### Details

**Sources** - [[Bug]: ValueError: Tool with name python not found](https://github.com/run-llama/llama_index/issues/12479) - [[Bug]: Agent sometimes tries to use tools that do not exist and then crashes](https://github.com/run-llama/llama_index/issues/7170) - [llama-index-integrations/agent/llama-index-agent-openai-legacy/llama_index/agent/openai_legacy/utils.py](https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/agent/llama-index-agent-openai-legacy/llama_index/agent/openai_legacy/utils.py) - [llama-index-integrations/agent/llama-index-agent-openai/llama_index/agent/openai/utils.py](https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/agent/llama-index-agent-openai/llama_index/agent/openai/utils.py) - [llama-index-legacy/llama_index/legacy/tools/__init__.py](https://github.com/run-llama/llama_index/blob/main/llama-index-legacy/llama_index/legacy/tools/__init__.py) - [llama-index-core/llama_index/core/tools/__init__.py](https://github.com/run-llama/llama_index/blob/main/llama-index-core/llama_index/core/tools/__init__.py) - [llama-index-core/llama_index/core/tools/query_engine.py](https://github.com/run-llama/llama_index/blob/main/llama-index-core/llama_index/core/tools/query_engine.py) - [llama-index-legacy/llama_index/legacy/tools/query_engine.py](https://github.com/run-llama/llama_index/blob/main/llama-index-legacy/llama_index/legacy/tools/query_engine.py) - [llama-index-legacy/llama_index/legacy/langchain_helpers/agents/tools.py](https://github.com/run-llama/llama_index/blob/main/llama-index-legacy/llama_index/legacy/langchain_helpers/agents/tools.py) - [llama-index-integrations/agent/llama-index-agent-openai-legacy/llama_index/agent/openai_legacy/openai_agent.py](https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/agent/llama-index-agent-openai-legacy/llama_index/agent/openai_legacy/openai_agent.py) - [llama-index-core/tests/indices/tree/test_retrievers.py](https://github.com/run-llama/llama_index/blob/main/llama-index-core/tests/indices/tree/test_retrievers.py) - [llama-index-legacy/tests/indices/vector_store/test_retrievers.py](https://github.com/run-llama/llama_index/blob/main/llama-index-legacy/tests/indices/vector_store/test_retrievers.py)
**About Dosu** This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

Hi @dosu, I have verified my tools names I am using in my queries match the names defined in the ToolMetadata.

But the error still happend.

To resolve the ValueError: Tool with name functions not found error, despite verifying that the tool names in your queries match the names defined in the ToolMetadata, you should consider the following steps and checks:

Step-by-Step Debugging

Print Available Tool Names: Modify the get_function_by_name function to print the available tool names before raising the error. This will help you identify if the tool names are being correctly registered.

def get_function_by_name(tools: List[BaseTool], name: str) -> BaseTool:
   """Get function by name."""
   name_to_tool = {tool.metadata.name: tool for tool in tools}
   print("Available tools:", name_to_tool.keys())  # Add this line for debugging
   if name not in name_to_tool:
       raise ValueError(f"Tool with name {name} not found")
   return name_to_tool[name]

Check for Duplicates and Correct Initialization: Ensure that each tool is uniquely named and properly initialized. Here is an example of how you might initialize QueryEngineTool objects:

from llama_index.core.tools.query_engine import QueryEngineTool
from llama_index.core.query_engine.router_query_engine import ToolMetadata

tools = []
for file_name in file_names:
   metadata = ToolMetadata(name=file_name, description=f"Tool for {file_name}")
   tool = QueryEngineTool(query_engine=some_query_engine, metadata=metadata)
   tools.append(tool)

Verify Tool Names in Queries: Ensure that the tool names used in your queries exactly match the names defined in the ToolMetadata. This includes checking for any typos, case sensitivity, and leading/trailing spaces.

Check Tool Retrieval Logic: Ensure that the logic for retrieving tools by name is correctly implemented. Here is an example of how you might retrieve a tool by name:

tool_name = "desired_tool_name"
try:
   tool = get_function_by_name(tools, tool_name)
   print(f"Successfully retrieved tool: {tool.metadata.name}")
except ValueError as e:
   print(e)

Example Code for Initialization and Retrieval

Here is a complete example that includes tool initialization, printing available tool names, and retrieving a tool by name:

from llama_index.core.tools.query_engine import QueryEngineTool
from llama_index.core.query_engine.router_query_engine import ToolMetadata
from typing import List

# Define the get_function_by_name function with debugging print statement
def get_function_by_name(tools: List[BaseTool], name: str) -> BaseTool:
    """Get function by name."""
    name_to_tool = {tool.metadata.name: tool for tool in tools}
    print("Available tools:", name_to_tool.keys())  # Add this line for debugging
    if name not in name_to_tool:
        raise ValueError(f"Tool with name {name} not found")
    return name_to_tool[name]

# Initialize tools
tools = []
file_names = ["file1", "file2", "file3"]
for file_name in file_names:
    metadata = ToolMetadata(name=file_name, description=f"Tool for {file_name}")
    tool = QueryEngineTool(query_engine=some_query_engine, metadata=metadata)
    tools.append(tool)

# Attempt to retrieve a tool by name
tool_name = "file1"
try:
    tool = get_function_by_name(tools, tool_name)
    print(f"Successfully retrieved tool: {tool.metadata.name}")
except ValueError as e:
    print(e)

By following these steps and ensuring that the tool names are correctly defined, initialized, and retrieved, you should be able to identify and correct the mismatch causing the ValueError. If the issue persists, please provide additional details or code snippets for further assistance.

#### Details

**Sources** - [llama-index-legacy/llama_index/legacy/agent/openai/utils.py](https://github.com/run-llama/llama_index/blob/main/llama-index-legacy/llama_index/legacy/agent/openai/utils.py) - [llama-index-integrations/agent/llama-index-agent-openai-legacy/llama_index/agent/openai_legacy/utils.py](https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/agent/llama-index-agent-openai-legacy/llama_index/agent/openai_legacy/utils.py) - [llama-index-integrations/agent/llama-index-agent-openai/llama_index/agent/openai/utils.py](https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/agent/llama-index-agent-openai/llama_index/agent/openai/utils.py) - [llama-index-core/llama_index/core/query_engine/router_query_engine.py](https://github.com/run-llama/llama_index/blob/main/llama-index-core/llama_index/core/query_engine/router_query_engine.py) - [llama-index-core/llama_index/core/tools/query_engine.py](https://github.com/run-llama/llama_index/blob/main/llama-index-core/llama_index/core/tools/query_engine.py) - [llama-index-legacy/llama_index/legacy/tools/query_engine.py](https://github.com/run-llama/llama_index/blob/main/llama-index-legacy/llama_index/legacy/tools/query_engine.py) - [llama-index-core/tests/objects/test_base.py](https://github.com/run-llama/llama_index/blob/main/llama-index-core/tests/objects/test_base.py) - [llama-index-core/tests/indices/vector_store/test_simple.py](https://github.com/run-llama/llama_index/blob/main/llama-index-core/tests/indices/vector_store/test_simple.py) - [llama-index-legacy/tests/objects/test_base.py](https://github.com/run-llama/llama_index/blob/main/llama-index-legacy/tests/objects/test_base.py) - [llama-index-legacy/llama_index/legacy/agent/legacy/openai_agent.py](https://github.com/run-llama/llama_index/blob/main/llama-index-legacy/llama_index/legacy/agent/legacy/openai_agent.py) - [llama-index-integrations/agent/llama-index-agent-openai-legacy/llama_index/agent/openai_legacy/openai_agent.py](https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/agent/llama-index-agent-openai-legacy/llama_index/agent/openai_legacy/openai_agent.py) - [llama-index-legacy/llama_index/legacy/agent/openai/base.py](https://github.com/run-llama/llama_index/blob/main/llama-index-legacy/llama_index/legacy/agent/openai/base.py) - [llama-index-integrations/agent/llama-index-agent-openai/llama_index/agent/openai/base.py](https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/agent/llama-index-agent-openai/llama_index/agent/openai/base.py) - [llama-index-core/llama_index/core/tools/retriever_tool.py](https://github.com/run-llama/llama_index/blob/main/llama-index-core/llama_index/core/tools/retriever_tool.py)
**About Dosu** This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

I'm not sure what LLM you are using, but it's just hallucinating a tool name, which can happen

Hi @logan-markewich, I use AzuerOpenAI gpt3.5, so could this be an issue with the LLM API?

Hi @dosu , some queries won't result in this error, but some querirs will happen..

this won't happen

this happen

To address the intermittent ValueError: Tool with name functions not found error when using the top_agent.query() function, follow these steps to ensure correct tool name recognition and mitigate potential issues with the LLM API hallucinating tool names:

Steps to Ensure Correct Tool Name Recognition

Verify Tool Initialization and Retrieval: Ensure that the tools are correctly initialized and their names are properly set. Add detailed logging to trace the issue more effectively.
Enhance Logging for Debugging: Add detailed logging to trace the available tools and the requested tool name. This will help identify if the tool names are being correctly passed and recognized.
Check for LLM API Hallucinations: Since the issue might be related to the LLM API hallucinating tool names, add additional checks and logging to capture the exact tool names being requested by the LLM.

Enhanced Code with Detailed Logging

`get_function_by_name` Function with Enhanced Logging

import logging

logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)  # Set to DEBUG to capture detailed logs

def get_function_by_name(tools: List[BaseTool], name: str) -> BaseTool:
    """Get function by name."""
    name_to_tool = {tool.metadata.name: tool for tool in tools}
    logger.debug(f"Available tools: {list(name_to_tool.keys())}")
    logger.debug(f"Requested tool name: {name}")
    if name not in name_to_tool:
        logger.error(f"Tool with name {name} not found")
        raise ValueError(f"Tool with name {name} not found")
    return name_to_tool[name]

Tool Initialization with Logging

from llama_index.core.tools.query_engine import QueryEngineTool
from llama_index.core.query_engine.router_query_engine import ToolMetadata

tools = []
file_names = ["file1", "file2", "file3"]
for file_name in file_names:
    metadata = ToolMetadata(name=file_name, description=f"Tool for {file_name}")
    tool = QueryEngineTool(query_engine=some_query_engine, metadata=metadata)
    tools.append(tool)
    logger.debug(f"Initialized tool: {metadata.name}")

Tool Retrieval with Logging

tool_name = "file1"
try:
    tool = get_function_by_name(tools, tool_name)
    logger.debug(f"Successfully retrieved tool: {tool.metadata.name}")
except ValueError as e:
    logger.error(e)

Additional Debugging Mechanisms

Callback Handler: Use the callback handler provided by LlamaIndex to trace the inner workings of the library.
```
import llama_index.core

llama_index.core.set_global_handler("simple")
```
Observability: Integrate with observability tools to view LLM/prompt inputs/outputs and ensure component outputs are performing as expected.

Mitigating LLM API Hallucinations

If the issue persists and you suspect that the LLM API might be hallucinating tool names, consider the following:

Explicit Tool Name Validation: Before making a query, explicitly validate the tool name against the list of available tools.
Fallback Mechanism: Implement a fallback mechanism to handle cases where the tool name is not found. For example, you can log the error and return a default response instead of raising an exception.
Rate Limiting and Retries: Implement rate limiting and retries to handle intermittent issues with the LLM API.

By following these steps and enhancing your logging and debugging mechanisms, you should be able to trace the source of the ValueError: Tool with name functions not found error and mitigate potential issues with the LLM API hallucinating tool names.

#### Details

**Sources** - [llama-index-integrations/agent/llama-index-agent-openai-legacy/llama_index/agent/openai_legacy/openai_agent.py](https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/agent/llama-index-agent-openai-legacy/llama_index/agent/openai_legacy/openai_agent.py) - [llama-index-integrations/agent/llama-index-agent-openai-legacy/llama_index/agent/openai_legacy/utils.py](https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/agent/llama-index-agent-openai-legacy/llama_index/agent/openai_legacy/utils.py) - [llama-index-core/llama_index/core/tools/query_engine.py](https://github.com/run-llama/llama_index/blob/main/llama-index-core/llama_index/core/tools/query_engine.py) - [llama-index-legacy/llama_index/legacy/tools/query_engine.py](https://github.com/run-llama/llama_index/blob/main/llama-index-legacy/llama_index/legacy/tools/query_engine.py) - [llama-index-core/llama_index/core/agent/function_calling/step.py](https://github.com/run-llama/llama_index/blob/main/llama-index-core/llama_index/core/agent/function_calling/step.py) - [llama-index-legacy/llama_index/legacy/agent/openai/step.py](https://github.com/run-llama/llama_index/blob/main/llama-index-legacy/llama_index/legacy/agent/openai/step.py) - [docs/docs/understanding/tracing_and_debugging/tracing_and_debugging.md](https://github.com/run-llama/llama_index/blob/main/docs/docs/understanding/tracing_and_debugging/tracing_and_debugging.md) - [llama-index-integrations/agent/llama-index-agent-openai/llama_index/agent/openai/step.py](https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/agent/llama-index-agent-openai/llama_index/agent/openai/step.py)
**About Dosu** This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

run-llama / llama_index