SQL Database Agent - 'Is not a valid tool' error

problemxl commented 4 months ago

Checked other resources

[X] I added a very descriptive title to this issue.
[X] I searched the LangChain documentation with the integrated search.
[X] I used the GitHub search to find a similar question and didn't find it.
[X] I am sure that this is a bug in LangChain rather than my code.
[X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

The following code:

from langchain_community.utilities import SQLDatabase

from langchain.agents import create_sql_agent
from langchain_community.agent_toolkits import SQLDatabaseToolkit
from langchain.agents.agent_types import AgentType
from langchain_groq import ChatGroq

llm = ChatGroq(temperature=0.5, groq_api_key="<api-key>", model_name="mixtral-8x7b-32768")
db = SQLDatabase.from_uri("sqlite:///Chinook.db")

agent_executor_kwargs = {
        "handle_parsing_errors": True,
    }

agent_executor = create_sql_agent(
        llm=llm,
        toolkit=SQLDatabaseToolkit(db=db, llm=llm),
        # db=db,
        verbose=True,
        agent_type=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
        agent_executor_kwargs=agent_executor_kwargs,
    )

agent_executor.invoke(
    {"input": "List the total sales per country. Which country's customers spent the most?"}
)

Error Message and Stack Trace (if applicable)

> Entering new SQL Agent Executor chain...
I need to query the database to get the total sales per country. After that, I can identify the country whose customers spent the most.

Action: sql\_db\_query
Action Input: SELECT country, SUM(sales) FROM orders GROUP BY country ORDER BY SUM(sales) DESC;
sql\_db\_query is not a valid tool, try one of [sql_db_query, sql_db_schema, sql_db_list_tables, sql_db_query_checker].I apologize for the confusion. You are correct that the action should be "sql\_db\_query". However, it seems that I made a mistake in the input for the action. I will try again.

Action: sql\_db\_query
Action Input: SELECT country, SUM(sales) FROM orders GROUP BY country ORDER BY SUM(sales) DESC;
sql\_db\_query is not a valid tool, try one of [sql_db_query, sql_db_schema, sql_db_list_tables, sql_db_query_checker].I apologize for the confusion. It seems I need to use the `sql_db_query_checker` tool first to ensure that my query is correct before executing it with `sql_db_query`.

Action: sql\_db\_query\_checker
Action Input: SELECT country, SUM(sales) FROM orders GROUP BY country ORDER BY SUM(sales) DESC;
sql\_db\_query\_checker is not a valid tool, try one of [sql_db_query, sql_db_schema, sql_db_list_tables, sql_db_query_checker].I apologize for the confusion. It seems I need to check the table schema first to ensure that my query is correct. I will use `sql_db_schema` tool to get the schema of the `orders` table.

Action: sql\_db\_schema
Action Input: orders
sql\_db\_schema is not a valid tool, try one of [sql_db_query, sql_db_schema, sql_db_list_tables, sql_db_query_checker].I apologize for the confusion. It seems I made a mistake in the input for the action. The correct tool to use is `sql_db_list_tables` to get the list of tables in the database.

Action: sql\_db\_list\_tables
Action Input: 
sql\_db\_list\_tables is not a valid tool, try one of [sql_db_query, sql_db_schema, sql_db_list_tables, sql_db_query_checker].I apologize for the confusion. I realize now that I made a mistake in my previous inputs. The correct tool to use is `sql_db_list_tables` to get the list of tables in the database. I will try again.
...
Action Input: 
sql\_db\_list\_tables is not a valid tool, try one of [sql_db_query, sql_db_schema, sql_db_list_tables, sql_db_query_checker].Invalid Format: Missing 'Action:' after 'Thought:Invalid Format: Missing 'Action:' after 'Thought:Invalid Format: Missing 'Action:' after 'Thought:Invalid Format: Missing 'Action:' after 'Thought:Invalid Format: Missing 'Action:' after 'Thought:Invalid Format: Missing 'Action:' after 'Thought:Invalid Format: Missing 'Action:' after 'Thought:Invalid Format: Missing 'Action:' after 'Thought:Invalid Format: Missing 'Action:' after 'Thought:

> Finished chain.

Description

When attempting to use the SQL Database agent with LLMs that aren't from OpenAI, it produces this loop caused by back slash escape sequences.

I have included groq utilizing the mixtral model but I have also experienced this using ollama, llama.cpp testing a number of different LLM models including but not limited to mixtral, mistral, llama, phi, codellama, etc.

I do not experience this bug when using gpt-3.5-turbo and gpt-4

System Info

langchain: 0.1.9 python: 3.11.3 OS: Windows

keenborder786 commented 4 months ago

Can you please update your langchain version to latest and try again. That will resolve the issue.

problemxl commented 4 months ago

I updated to langchain 0.1.10 and I'm experiencing the same issue.

codeloki15 commented 4 months ago

Same issue , what I would suggest is the usage of agent type.. When I am using gpt3.5 turbo with open ai function on same db .. it is retrieving the results When I am using groq, we canyt use this function and looks like zero shot is not the way forwards

codeloki15 commented 4 months ago

#llm = ChatOpenAI(model="gpt-3.5-turbo", temperature=0)
agent_executor = create_sql_agent(llm, db=db, agent_type="zero-shot-react-description", verbose=True)

Entering new SQL Agent Executor chain... To answer this question, I need to first understand the structure of the "anime_reviwes" table. I will use the "sql_db_schema" tool to get the schema and sample rows of the table.

Action: sql_db_schema Action Input: anime_reviews sql_db_schema is not a valid tool, try one of [sql_db_query, sql_db_schema, sql_db_list_tables, sql_db_query_checker].I made a mistake in using the tool's name. I should use "sql_db_schema" to get the schema of the "anime_reviews" table.

Action: sql_db_schema Action Input: anime_reviews sql_db_schema is not a valid tool, try one of [sql_db_query, sql_db_schema, sql_db_list_tables, sql_db_query_checker].I made a mistake in the tool name, it should be "sql_db_schema". I will try again.

Action: sql_db_schema Action Input: anime_reviews sql_db_schema is not a valid tool, try one of [sql_db_query, sql_db_schema, sql_db_list_tables, sql_db_query_checker].I am facing an issue where the provided tool "sql_db_schema" is not being recognized. I will inform the user about this issue so they can take appropriate action.

Final Answer: It seems there is an issue with the provided tools as "sql_db_schema" is not being recognized. Please check the tool availability or notify the system administrator.

Finished chain. {'input': 'List the top animes in anime_reviwes table ?', 'output': 'It seems there is an issue with the provided tools as "sql\_db\_schema" is not being recognized. Please check the tool availability or notify the system administrator.'}

spagliarini commented 3 months ago

Did anyone found a solution to this issue?

I updated to langchain 0.1.12 and I'm experiencing the same issue.

linuxmanju commented 3 months ago

Looks mostly to do with Mixtral escaping "" with "_" causing the issue. in my local llama I tried system prompt 'Do not escape "" OR underscores' Works many times not always, may be you can give it a try.

machuck commented 3 months ago

version 0.1.13 same issue here

michaelgloeckner commented 2 months ago

As a workaround you can make a copy of the SQLDatabaseToolkit and use it with modified names.

copy lib\python\site-package\langchain_community\agen_toolkits\sql\toolkit.py and lib\python\site-package\langchain_community\tools\sql_database\tool.py in workspace.
use tool.py copy in toolkit.py

from tool import (
    InfoSQLDatabaseTool,
    ListSQLDatabaseTool,
    QuerySQLCheckerTool,
    QuerySQLDataBaseTool,
)

change names of the tool names in tool.py

eg.: name: str = "sqldbschema" # instead of sql_db_schema

in your implementation use:

from toolkit import SQLDatabaseToolkit

yalvhe2009 commented 5 days ago

As a workaround you can make a copy of the SQLDatabaseToolkit and use it with modified names.

copy lib\python\site-package\langchain_community\agen_toolkits\sql\toolkit.py and lib\python\site-package\langchain_community\tools\sql_database\tool.py in workspace.

use tool.py copy in toolkit.py
from tool import (
    InfoSQLDatabaseTool,
    ListSQLDatabaseTool,
    QuerySQLCheckerTool,
    QuerySQLDataBaseTool,
)
change names of the tool names in tool.py

eg.: name: str = "sqldbschema" # instead of sql_db_schema

in your implementation use:

from toolkit import SQLDatabaseToolkit

Thanks, your method works.

langchain-ai / langchain