AgentExecutor is not choosing the tool (vanna_tool) when using any Gemini model (it is working properly when using OpenAI or Mistral)

mkhansa commented 1 week ago

Checked other resources

[X] I added a very descriptive title to this issue.
[X] I searched the LangChain documentation with the integrated search.
[X] I used the GitHub search to find a similar question and didn't find it.
[X] I am sure that this is a bug in LangChain rather than my code.
[X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

class MyVanna(ChromaDB_VectorStore, GoogleGeminiChat): def init(self, config=None): ChromaDB_VectorStore.init(self, config=config) GoogleGeminiChat.init(self, config=config)

vn = MyVanna(config={ "path": "../...., "api_key": os.getenv("GOOGLE_API_KEY"), "model_name": "gemini-1.5-pro", } )

@tool('vanna_tool') def vanna_tool(qa : str): "....." .... return ..... class Response(BaseModel): """The format for your final response by the agent should be in a Json format agent_response and sql_query"""

agent_response: str = Field(description="""..... """)
sql_query: str = Field("", description="The full sql query returned by the `vanna_tool` agent and '' if no response.")

tools = [vanna_tool]

llm = ChatVertexAI(model="gemini-pro")

...

parser = PydanticOutputParser(pydantic_object=Response) prompt.partial_variables = {"format_instructions": parser.get_format_instructions()}

agent = create_tool_calling_agent(llm, tools, prompt)

agent_executor = AgentExecutor( agent=agent, tools=tools, verbose=True, return_intermediate_steps=True )

Error Message and Stack Trace (if applicable)

in case of openai:


Invoking: `vanna_tool` with `{'qa': "Display the '...' column values for the week between 2 to 8 in 2024."}`
...
``` **(right output)**

in case of Gemini:

Entering new AgentExecutor chain...


{"agent_response": "Here are the ... values for weeks 2 to 8 in 2024:\n\n```\n                   opec  \nweek  \n2       77680000     \n3       77900000     \n4       78110000     \n5       78310000     \n6       78500000     \n7       78680000     \n```\n\nI hope this is helpful!", "sql_query": "SELECT ...FROM ...WHERE week BETWEEN 2 AND 8 AND YEAR_ = 2024;"}
``` **(random output without invoking 'vanna_tool'**

Description

asking question that should invoke vanna_tool when using openai or mistral, it works. when using any gemini model, it will create random response without entering 'vanna_tool' ..

System Info

langchain==0.2.5 langchain-community==0.2.5 langchain-core==0.2.9 langchain-experimental==0.0.61 langchain-openai==0.1.9

langchain-google-genai==1.0.6 langchain-google-vertexai==1.0.5 google-generativeai==0.5.4

Animeshkr9044 commented 1 week ago

what is the python version you are using ?

mkhansa commented 1 week ago

what is the python version you are using ?

Python 3.12.2, windows

langchain-ai / langchain