[tool_name] is not a valid tool, try one of [tool_name]

[Raji0196]

Issue you'd like to raise.

I am using Opensearch as vector database in langchain to store the documents and using tools to integrate it with agent to look for the information when we pass any query and to use GPT model to get the enhanced response.

I am facing tool not valid issue while running the code. Any suggestions to fix the error.

Error: Observation:[faq] is not a valid tool, try one of [faq].

Code:

import re from langchain import OpenAI, PromptTemplate, VectorDBQA, LLMChain from langchain.agents import Tool, initialize_agent, AgentExecutor, AgentOutputParser, LLMSingleActionAgent from langchain.embeddings import OpenAIEmbeddings from langchain.chains import ConversationChain, RetrievalQA from langchain.chains.conversation.memory import ConversationBufferMemory, ConversationSummaryBufferMemory, CombinedMemory import pypdf from langchain.prompts import StringPromptTemplate from langchain.schema import HumanMessage, AgentAction, AgentFinish from langchain.text_splitter import CharacterTextSplitter from langchain.tools import BaseTool from langchain.memory import ConversationBufferWindowMemory import os from typing import List, Union, Optional from langchain.memory import ConversationSummaryBufferMemory from langchain.vectorstores import Chroma, OpenSearchVectorSearch

os.environ['OPENAI_API_KEY'] = "api_key"

embeddings = OpenAIEmbeddings() llm = OpenAI(temperature=0.7) docsearch = OpenSearchVectorSearch( index_name="customer_data", embedding_function=embeddings, opensearch_url="host_url", )

qa = RetrievalQA.from_chain_type( llm=llm, chain_type="stuff", retriever=docsearch.as_retriever(), )

tools = [ Tool( name="faq", func=qa.run(), description="Useful when you have to answer FAQ's" )]

template = """Your are a support representative, Customers reaches you for queries. refer the tools and answer. You have access to the following tools to answer the question. {tools}

Use the following format:

Question: the input question you must answer Thought: you should always think about what to do Action: the action to take, can be one of [{tool_names}] Action Input: the input to the action Observation: the result of the action ... (this Thought/Action/Action Input/Observation can repeat N times) Thought: I now know the final answer Final Answer: the final answer to the original input question.

Begin! Previous conversation history: {history}

New Question: {input} {agent_scratchpad}"""

Set up a prompt template

class CustomPromptTemplate(StringPromptTemplate):

The template to use

template: str
# The list of tools available
tools: List[Tool]

def format(self, **kwargs) -> str:
    # Get the intermediate steps (AgentAction, Observation tuples)
    # Format them in a particular way
    intermediate_steps = kwargs.pop("intermediate_steps")
    thoughts = ""
    for action, observation in intermediate_steps:
        thoughts += action.log
        thoughts += f"\nObservation: {observation}\nThought: "
    # Set the agent_scratchpad variable to that value
    kwargs["agent_scratchpad"] = thoughts
    # Create a tools variable from the list of tools provided
    kwargs["tools"] = "\n".join([f"{tool.name}: {tool.description}" for tool in self.tools])
    # Create a list of tool names for the tools provided
    kwargs["tool_names"] = ", ".join([tool.name for tool in self.tools])
    return self.template.format(**kwargs)

prompt = CustomPromptTemplate( template=template, tools=tools,

This omits the agent_scratchpad, tools, and tool_names variables because those are generated dynamically

# This includes the `intermediate_steps` variable because that is needed
input_variables=["input", "intermediate_steps", "history"]

)

Custom Output Parser. The output parser is responsible for parsing the LLM output into AgentAction and AgentFinish. This usually depends heavily on the prompt used. This is where you can change the parsing to do retries, handle whitespace, etc

class CustomOutputParser(AgentOutputParser):

def parse(self, llm_output: str) -> Union[AgentAction, AgentFinish]:
    # Check if agent should finish
    if "Final Answer:" in llm_output:
        return AgentFinish(
            # Return values is generally always a dictionary with a single `output` key
            # It is not recommended to try anything else at the moment :)
            return_values={"output": llm_output.split("Final Answer:")[-1].strip()},
            log=llm_output,
        )
    # Parse out the action and action input
    regex = r"Action\s*\d*\s*:(.*?)\nAction\s*\d*\s*Input\s*\d*\s*:[\s]*(.*)"
    match = re.search(regex, llm_output, re.DOTALL)
    print(match)
    if not match:
        raise ValueError(f"Could not parse LLM output: `{llm_output}`")
    action = match.group(1).strip()
    action_input = match.group(2)
    # Return the action and action input
    return AgentAction(tool=action, tool_input=action_input.strip(" ").strip('"'), log=llm_output)

output_parser = CustomOutputParser()

Set up LLM !

llm = OpenAI(temperature=0, model_name='gpt-3.5-turbo')

Define the stop sequence. This is important because it tells the LLM when to stop generation. This depends heavily on the prompt and model you are using. Generally, you want this to be whatever token you use in the prompt to denote the start of an Observation (otherwise, the LLM may hallucinate an observation for you).

Set up the Agent

LLM chain consisting of the LLM and a prompt

llm_chain = LLMChain(llm=llm, prompt=prompt)

tool_names = [tool.name for tool in tools]

agent = LLMSingleActionAgent( llm_chain=llm_chain, output_parser=output_parser, stop=["\nObservation:"], allowed_tools=tool_names )

Agent Executors take an agent and tools and use the agent to decide which tools to call and in what order.

convo_memory = ConversationBufferMemory( memory_key="chat_history_lines", input_key="input" )

summary_memory = ConversationSummaryBufferMemory(llm=llm, memory_key="chat_history", input_key="input")

memory = CombinedMemory(memories=[convo_memory, summary_memory], memory_key="story")

agent_executor = AgentExecutor.from_agent_and_tools( agent=agent, tools=tools, verbose=True, memory=memory )

si = input("Human: ") agent_executor.run({'history': memory, 'input': si})

Suggestion:

No response

[dosubot[bot]]

🤖

Hello,

The error message you're seeing, "Observation:[faq] is not a valid tool, try one of [faq]", suggests that the tool you're trying to use, "faq", is not recognized as a valid tool in your current setup. This could be due to a few reasons:

1. The tool is not properly defined or initialized.
2. The tool is not included in the list of allowed tools for the agent.

Looking at your code, it seems like you've defined the "faq" tool and included it in the list of tools for the agent. However, the error message suggests that the tool is not recognized when it's being used.

Here are a few things you can check:

1. Tool Initialization: Make sure the tool is properly initialized. In your code, you're initializing the "faq" tool with qa.run(). However, run() is a method that should be called when the tool is used, not when it's initialized. Try changing this to just qa:

tools = [
    Tool(
        name="faq",
        func=qa,
        description="Useful when you have to answer FAQ's"
    )]

2. Allowed Tools: Make sure the tool is included in the list of allowed tools for the agent. In your code, you're creating a list of tool names and passing it to the LLMSingleActionAgent. Make sure the tool names match exactly with the names you've defined for the tools. In your case, it should be "faq".

3. Tool Execution: Make sure the tool is properly executed when it's used. In your code, you're using the AgentExecutor to run the agent with the input. Make sure the tool is properly called and executed in this process.

If you've checked all these and the issue still persists, it would be helpful if you could provide more information about the error. Specifically, it would be helpful to know:

• The exact line of code where the error occurs.
• The full error message and stack trace.
• Any additional context or information about your setup.

This will help us better understand the issue and provide a more accurate solution.

I hope this helps! Let me know if you have any other questions.

Best, Dosu

Sources

• Issue: The agent with the text-davinci-003 model and serpapi tool attempted to utilize a non-existent tool
• libs/langchain/langchain/agents/load_tools.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[dosubot[bot]]

Hi, @Raji0196,

I'm helping the LangChain team manage their backlog and am marking this issue as stale. From what I understand, the issue you raised was related to a tool not being recognized as valid when running the code. Dosubot provided a detailed response suggesting potential reasons for the error and steps to troubleshoot, including checking the tool initialization, allowed tools, and tool execution. It seems that the issue has been resolved based on the detailed response provided.

Could you please confirm if this issue is still relevant to the latest version of the LangChain repository? If it is, please let the LangChain team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or the issue will be automatically closed in 7 days.

Thank you for your understanding and contribution to LangChain!

[VivekLokesh]

Don't close the issue please. I'm experiencing the same issue as well. The tools are properly defined in my code but the agent observation process is continuously looping on the tools thinking the provided tools are not valid.

Observation: the tool to use, should be one of [tool1, tool2, tool3, tool4, tool5 ] is not a valid tool, try one of [tool1, tool2, tool3, tool4, tool5 ].

Posting a portion of the code here for review, just masking some sensitive details with generic names like tool 1 and purpose 1 and so on.

# custom prompt template

PREFIX = '''<custom system prompt about the agent's persona and purpose>
'''

FORMAT_INSTRUCTIONS = """To use a tool, please use the following format:
'''
Thought: you should always think about what to do
Action: the tool to use, should be one of [{tool_names}]
Action Input: the input to the tool you selected
Observation: the result of the tool 
... (this Thought/Action/Action Input/Observation can repeat N times)
Thought: I now know the final answer
Final Answer: the final answer to the original input question
'''
"""

SUFFIX = '''
Begin!

Instructions: {input}
{agent_scratchpad}
'''

tools = [
    Tool(
        name="tool 1",
        func=qa_chain.run,  #qa_chain is a llm chain defined in the code earlier.
        description="use this tool for purpose 1"
    ),
    Tool(
        name="tool 2",
        func=tool2,
        description="use this tool for purpose 2"
    ),
    Tool(
        name="tool 3",
        func=tool3,
        description="use this tool for purpose 3"
    ),
    Tool(
        name="tool 4",
        func=tool4,
        description="use this tool for purpose 4"
    ),
    Tool(
        name="tool 5",
        func=tool5,
        description="use this tool for purpose 5"

    )    
]

agent = initialize_agent(
    tools=tools,
    llm=sm_llm,
    agent=AgentType.CONVERSATIONAL_REACT_DESCRIPTION,
    verbose=True,
    max_iterations=4,
    handle_parsing_errors=True,
    early_stopping_method='generate',
    memory=memory,
    agent_kwargs={
        'prefix': PREFIX, 
        'format_instructions': FORMAT_INSTRUCTIONS,
        'suffix': SUFFIX
        }
)

Output:

Thought: you should always think about what to do
Action: the tool to use, should be one of [tool1, tool2, tool3, tool4, tool5 ]
Action Input: the input to the tool you selected
Observation: the tool to use, should be one of [tool1, tool2, tool3, tool4, tool5 ] is not a valid tool, try one of [tool1, tool2, tool3, tool4, tool5 ].
Thought: <and it executes my system prompt here for some reason>

has anyone else ran into this behavior and overcame it? fyi - @efriis @hinthornw @hwchase17

[LandJerry]

I encountered the same issue and resolved it; the problem mainly stemmed from the model not recognizing the tool. GPT-3.5 Turbo and GPT-4 have different levels of tool recognition. You can look at the section "# Otherwise we lookup the tool" in the _aperform_agent_action and _iter_next_step methods. I solved this problem by changing the Action part of the prompt template.

[ZuzooVn]

@LandJerry

I solved this problem by changing the Action part of the prompt template.

Could you please share more details about your solution?

[jonmach]

I'm also having the same issue @LandJerry - could you please share more details?