Open gavinslim opened 1 month ago
Hello @gavinslim! I'm here to help you with any bugs, questions, or contributions you might have. Let's work together to get things sorted out!
To display additional details on the tools being used by the top agent, you can use the verbose=True
parameter when creating the agent. This will provide more detailed output about the tools being used.
Here is an example of how to set this up:
from llama_index.core.agent import ReActAgent
from llama_index.llms.azure_openai import AzureOpenAI
from llama_index.core.tools import FunctionTool
from llama_index.tools.yahoo_finance import YahooFinanceToolSpec
# Define your tools
def multiply(a: float, b: float) -> float:
"""Multiply two numbers and returns the product"""
return a * b
multiply_tool = FunctionTool.from_defaults(fn=multiply)
def add(a: float, b: float) -> float:
"""Add two numbers and returns the sum"""
return a + b
add_tool = FunctionTool.from_defaults(fn=add)
# Fetch tools from LlamaHub
finance_tools = YahooFinanceToolSpec().to_tool_list()
finance_tools.extend([multiply_tool, add_tool])
# Create the agent with verbose=True
agent = ReActAgent.from_tools(finance_tools, verbose=True)
response = agent.chat("What is the current price of NVDA?")
print(response)
The verbose=True
parameter will ensure that additional details about the tools are displayed during the agent's operation. This is not dependent on whether you are using OpenAI or AzureOpenAI [1][2][3].
To continue talking to Dosu, mention @dosu.
Thanks for the reply. I have the verbose parameter set to True when creating the top agent, along with the agent for each document. See the code below. Is there anything else I should consider?
# Create top agent
top_agent = OpenAIAgent.from_tools(
tool_retriever=obj_index.as_retriever(similarity_top_k=3),
system_prompt=""" \
You are an agent designed to answer queries about a set of given cities.
Please always use the tools provided to answer a question. Do not rely on prior knowledge.\
""",
verbose=True,
)
To ensure that the additional details on the tools being used by the top agent are displayed, you should consider the following:
Tool Name and Description: Ensure the tool name and description are well-defined and informative. This helps the LLM understand the purpose and usage of the tool better.
tool = FunctionTool.from_defaults(
get_weather,
name="WeatherTool",
description="A tool for getting the weather for a given location."
)
Verbose Mode: Enable verbose mode when creating the agent, which you have already done.
agent = ReActAgent.from_tools(tools, llm=llm, verbose=True)
Return Direct Option: Consider using the return_direct
option if you want the response from the tool to be returned directly without being interpreted and rewritten by the agent.
tool = QueryEngineTool.from_defaults(
query_engine,
name="QueryEngineTool",
description="A tool for querying the engine.",
return_direct=True,
)
agent = OpenAIAgent.from_tools([tool])
Debugging Tools: Use debugging tools to inspect the tool schema and ensure it is correctly defined.
schema = tool.metadata.get_parameters_dict()
print(schema)
Using Existing Tools from LlamaHub: Leverage existing tools from LlamaHub to expand the agent's capabilities. For example, you can use the Yahoo Finance tool to fetch financial data.
from llama_index.tools.yahoo_finance import YahooFinanceToolSpec
finance_tools = YahooFinanceToolSpec().to_tool_list()
finance_tools.extend([multiply_tool, add_tool])
agent = ReActAgent.from_tools(finance_tools, verbose=True)
response = agent.chat("What is the current price of NVDA?")
print(response)
Forcing Tool Usage: If you want the agent to always use a specific tool, you can pass only that tool to the agent. This ensures that the agent uses the specified tool for its responses.
tools.append(Tool(
name="Search",
func=search.run,
description="Always use this tool first!",
return_direct=True
))
agent_chain = initialize_agent(tools, llm, agent="conversational-react-description", memory=memory, verbose=True, return_intermediate_steps=True)
By considering these aspects, you can ensure that the additional details on the tools being used by the top agent are displayed effectively [1][2][3].
To continue talking to Dosu, mention @dosu.
@dosu, I'm encountering another issue. It seems like my tools are missing to_openai_tool
response = top_agent.query("Please explain the files you have access to")
File "C:\Users\gavinlim\AppData\Local\Programs\Python\Python310\lib\site-packages\llama_index\core\instrumentation\dispatcher.py", line 274, in wrapper
result = func(*args, **kwargs)
File "C:\Users\gavinlim\AppData\Local\Programs\Python\Python310\lib\site-packages\llama_index\core\base\base_query_engine.py", line 53, in query
query_result = self._query(str_or_query_bundle)
File "C:\Users\gavinlim\AppData\Local\Programs\Python\Python310\lib\site-packages\llama_index\core\callbacks\utils.py", line 41, in wrapper
return func(self, *args, **kwargs)
File "C:\Users\gavinlim\AppData\Local\Programs\Python\Python310\lib\site-packages\llama_index\core\base\agent\types.py", line 43, in _query
agent_response = self.chat(
File "C:\Users\gavinlim\AppData\Local\Programs\Python\Python310\lib\site-packages\llama_index\core\instrumentation\dispatcher.py", line 274, in wrapper
result = func(*args, **kwargs)
File "C:\Users\gavinlim\AppData\Local\Programs\Python\Python310\lib\site-packages\llama_index\core\callbacks\utils.py", line 41, in wrapper
return func(self, *args, **kwargs)
File "C:\Users\gavinlim\AppData\Local\Programs\Python\Python310\lib\site-packages\llama_index\core\agent\runner\base.py", line 644, in chat
chat_response = self._chat(
File "C:\Users\gavinlim\AppData\Local\Programs\Python\Python310\lib\site-packages\llama_index\core\instrumentation\dispatcher.py", line 274, in wrapper
result = func(*args, **kwargs)
File "C:\Users\gavinlim\AppData\Local\Programs\Python\Python310\lib\site-packages\llama_index\core\agent\runner\base.py", line 574, in _chat
cur_step_output = self._run_step(
File "C:\Users\gavinlim\AppData\Local\Programs\Python\Python310\lib\site-packages\llama_index\core\instrumentation\dispatcher.py", line 274, in wrapper
result = func(*args, **kwargs)
File "C:\Users\gavinlim\AppData\Local\Programs\Python\Python310\lib\site-packages\llama_index\core\agent\runner\base.py", line 411, in _run_step
cur_step_output = self.agent_worker.run_step(step, task, **kwargs)
File "C:\Users\gavinlim\AppData\Local\Programs\Python\Python310\lib\site-packages\llama_index\core\callbacks\utils.py", line 41, in wrapper
return func(self, *args, **kwargs)
File "C:\Users\gavinlim\AppData\Local\Programs\Python\Python310\lib\site-packages\llama_index\agent\openai\step.py", line 728, in run_step
return self._run_step(
File "C:\Users\gavinlim\AppData\Local\Programs\Python\Python310\lib\site-packages\llama_index\agent\openai\step.py", line 567, in _run_step
openai_tools = [tool.metadata.to_openai_tool() for tool in tools]
File "C:\Users\gavinlim\AppData\Local\Programs\Python\Python310\lib\site-packages\llama_index\agent\openai\step.py", line 567, in <listcomp>
openai_tools = [tool.metadata.to_openai_tool() for tool in tools]
AttributeError: 'dict' object has no attribute 'to_openai_tool'
To resolve the issue with your tools missing the to_openai_tool
attribute, ensure that the ToolMetadata
class includes the to_openai_tool
method. Here is the relevant part of the code:
@dataclass
class ToolMetadata:
description: str
name: Optional[str] = None
fn_schema: Optional[Type[BaseModel]] = DefaultToolFnSchema
def get_parameters_dict(self) -> dict:
if self.fn_schema is None:
parameters = {
"type": "object",
"properties": {
"input": {"title": "input query string", "type": "string"},
},
"required": ["input"],
}
else:
parameters = self.fn_schema.schema()
parameters are {
k: v
for k, v in parameters.items()
if k in ["type", "properties", "required", "definitions"]
}
return parameters
@property
def fn_schema_str(self) -> str:
"""Get fn schema as string."""
if self.fn_schema is None:
raise ValueError("fn_schema is None.")
parameters = self.get_parameters_dict()
return json.dumps(parameters)
def get_name(self) -> str:
"""Get name."""
if self.name is None:
raise ValueError("name is None.")
return self.name
@deprecated(
"Deprecated in favor of `to_openai_tool`, which should be used instead."
)
def to_openai_function(self) -> Dict[str, Any]:
"""Deprecated and replaced by `to_openai_tool`.
The name and arguments of a function that should be called, as generated by the
model.
"""
return {
"name": self.name,
"description": self.description,
"parameters": self.get_parameters_dict(),
}
def to_openai_tool(self) -> Dict[str, Any]:
"""To OpenAI tool."""
return {
"type": "function",
"function": {
"name": self.name,
"description": self.description,
"parameters": self.get_parameters_dict(),
},
}
Ensure that your ToolMetadata
class includes the to_openai_tool
method as shown above. If your tools are instances of ToolMetadata
, they should have the to_openai_tool
attribute available [1][2][3].
To continue talking to Dosu, mention @dosu.
@dosu, My tools are not an object of ToolMetadata class. In step.py module, the returned tools from self.get_tools(task.input) returns a list of tools, each tool has a dict attribute called metadata that only has the name of the tool.
def _run_step(
self,
step: TaskStep,
task: Task,
mode: ChatResponseMode = ChatResponseMode.WAIT,
tool_choice: Union[str, dict] = "auto",
) -> TaskStepOutput:
"""Run step."""
if step.input is not None:
add_user_step_to_memory(
step, task.extra_state["new_memory"], verbose=self._verbose
)
# TODO: see if we want to do step-based inputs
# print(f'task.input: {task.input}')
tools = self.get_tools(task.input)
# print(f'Tools: {tools}')
# for tool in tools:
# print(f"DEBUG: {tool.metadata}")
# exit()
openai_tools = [tool.metadata.to_openai_tool() for tool in tools]
Question Validation
Question
I'm following the wikipedia multi-document agents example from: https://docs.llamaindex.ai/en/stable/examples/agent/multi_document_agents/. Instead of using OpenAI for the LLM, I am using AzureOpenAI.
In the wikipedia example, when a query is sent, "Calling function" descriptions are printed out along with the output to show which tool was called. However, when I perform the query with the top agent, I don't see this additional verbosity. Please see below for what I'm observing:
How do I display additional details on the tools that was using by the top agent? Is it because I'm not using OpenAI?
Thanks