Error creating AgentExecutor with custom Toolkit

[Punisheroot]

Hi, I am in the process of developing an agent and a toolkit to query mongodb databases, and to do this I simulated SQL Database Agent code with implementations for mongodb, I created this function to create the agent:

def create_mongodb_agent(
    llm: BaseLLM,
    toolkit: MongoDBDatabaseToolkit,
    callback_manager: Optional[BaseCallbackManager] = None,
    prefix: str = MONGODB_PREFIX,
    suffix: str = MONGODB_SUFFIX,
    format_instructions: str = FORMAT_INSTRUCTIONS,
    input_variables: Optional[List[str]] = None,
    top_k: int = 10,
    max_iterations: Optional[int] = 15,
    max_execution_time: Optional[float] = None,
    early_stopping_method: str = "force",
    verbose: bool = False,
    **kwargs: Any,
) -> AgentExecutor:
    """Construct a MongoDB agent from an LLM and tools."""
    tools = toolkit.get_tools()
    prefix = prefix.format(top_k=top_k)
    prompt = ZeroShotAgent.create_prompt(
        tools,
        prefix=prefix,
        suffix=suffix,
        format_instructions=format_instructions,
        input_variables=input_variables,
    )
    llm_chain = LLMChain(
        llm=llm,
        prompt=prompt,
        callback_manager=callback_manager,
    )
    tool_names = [tool.name for tool in tools]
    agent = ZeroShotAgent(llm_chain=llm_chain, allowed_tools=tool_names, **kwargs)
    return AgentExecutor.from_agent_and_tools(
        agent=agent,
        tools=tools,
        verbose=verbose,
        max_iterations=max_iterations,
        max_execution_time=max_execution_time,
        early_stopping_method=early_stopping_method,
    )

and this is my ToolKit:

class MongoDBDatabaseToolkit(BaseToolkit):
    """Toolkit for interacting with MongoDB databases."""

    db: MongoDBDatabase = Field(exclude=True)
    llm: BaseLLM = Field(default_factory=lambda: OpenAIChat(temperature=0))

    class Config:
        """Configuration for this pydantic object."""

        arbitrary_types_allowed = True

    def get_tools(self) -> List[BaseTool]:
        """Get the tools in the toolkit."""
        return [
            QueryMongoDBDatabaseTool(db=self.db),
            InfoMongoDBDatabaseTool(db=self.db),
            ListMongoDBDatabaseTool(db=self.db),
            QueryCheckerTool(db=self.db, llm=self.llm),
        ]

but I get the error from the AgentExecutor validator:

File "..\env\lib\site-packages\langchain\agents\initialize.py", line 64, in initialize_agent
    return AgentExecutor.from_agent_and_tools(
  File "..\env\lib\site-packages\langchain\agents\agent.py", line 557, in from_agent_and_tools
    return cls(
  File "pydantic\main.py", line 339, in pydantic.main.BaseModel.__init__
  File "pydantic\main.py", line 1102, in pydantic.main.validate_model
  File "..\env\lib\site-packages\langchain\agents\agent.py", line 565, in validate_tools
    tools = values["tools"]
KeyError: 'tools'

I did some debugging and verified that the tools are passed correctly to the AgentExecutor.from_agent_and_tools function. Can anyone help me with this?

[jeromeku]

Ran into similar problem.

The issue seems to be that BaseTool class subclasses pydantic BaseModel:

• During root validation phase of pydantic, the tool is not passing its check and therefore is not being passed to any methods marked @root_validator() in AgentExecutor, e.g., validate_tools, hence the KeyError. See here for more details on root validation.
• If you remove or comment out the @root_validator() decorators, you'll probably get a more informative message as to why pydantic validation is failing.
• For me, it was because my tool had an internal object that pydantic was complaining couldn't be pickled. Haven't figured out fix yet, but my guess is that you'll have to customize the Config of BaseTool.

[jeromeku]

@Punisheroot

Issue resolved for me. Try running your custom tool with the following env:

langchain==0.0.162
openapi-schema-pydantic==1.2.4
pydantic==1.10.7

[Punisheroot]

@Punisheroot

Issue resolved for me. Try running your custom tool with the following env:

langchain==0.0.162
openapi-schema-pydantic==1.2.4
pydantic==1.10.7

Hi @jeromeku i tried but still the same error 😢 The only one tool that have a root_validator is this:

class QueryCheckerTool(BaseMongoDBDatabaseTool, BaseTool):
    """Use an LLM to check if a MongoDB query is correct."""
    template: str = QUERY_CHECKER
    llm: BaseLLM
    llm_chain: LLMChain = Field(init=False)
    name = "query_checker_mongodb_db"
    description = """
    Use this tool to double check if your MongoDB query is correct before executing it.
    Always use this tool before executing a query with query_mongodb_db!
    """

    @root_validator(pre=True)
    def initialize_llm_chain(cls, values: Dict[str, Any]) -> Dict[str, Any]:
        if "llm_chain" not in values:
            values["llm_chain"] = LLMChain(
                llm=values.get("llm"),
                prompt=PromptTemplate(
                    template=QUERY_CHECKER, input_variables=["query"]
                ),
            )

        if values["llm_chain"].prompt.input_variables != ["query"]:
            raise ValueError(
                "LLM chain for QueryCheckerTool must have input variables ['query']"
            )

        return values

    def _run(self, query: str) -> str:
        """Use the LLM to check the MongoDB query."""
        return self.llm_chain.predict(query=query)

    async def _arun(self, query: str) -> str:
        return await self.llm_chain.apredict(query=query)

commenting root validator I get this error

Traceback (most recent call last):
  File "...\test.py", line 92, in <module>
    agent = create_mongodb_agent(llm=llm, toolkit=toolkit, verbose=True)
  File "...\mongodb\agent.py", line 32, in create_mongodb_agent
    tools = toolkit.get_tools()
  File "...\mongodb\toolkit.py", line 36, in get_tools
    QueryCheckerTool(db=self.db, llm=self.llm),
  File "pydantic\main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for QueryCheckerTool
llm_chain
  field required (type=value_error.missing)

[dosubot[bot]]

Hi, @Punisheroot! I'm Dosu, and I'm helping the LangChain team manage their backlog. I wanted to let you know that we are marking this issue as stale.

Based on my understanding of the current state of the issue, the problem is that the AgentExecutor validator is throwing a KeyError for the 'tools' key. You verified that the tools are being passed correctly to the AgentExecutor.from_agent_and_tools function. User jeromeku suggested that the issue may be related to the BaseTool class subclassing pydantic BaseModel and provided some suggestions for resolving the issue. However, you tried the suggested solution but still encountered the same error. You also provided additional details about the specific tool that has a root_validator and the error you received when commenting out the root_validator.

Now, we would like to know if this issue is still relevant to the latest version of the LangChain repository. If it is, please let us know by commenting on the issue. Otherwise, feel free to close the issue yourself or the issue will be automatically closed in 7 days.

Thank you for your understanding and cooperation!

[Zhreyu]

@Punisheroot How well did your custom chain work?