langchain-ai / langchain-google

MIT License
92 stars 108 forks source link

bug with ChatGoogleGenerativeAI: gemini can access only 1 tool although multiple tools given to it #369

Closed azliabdullah closed 1 month ago

azliabdullah commented 1 month ago

Bind a gemini model with multiple tools but it can only detect 1 tool (the first 1).

Compared with ChatOpenAi, it can access all tools given to it.

Regardless of how you invoke it either directly using model, chain or graph, or how you define the tools. I even tried using prebuilt tools. it only can detect 1 tool.

I'm using python 3.12.1 langchain-google-genai 1.0.7

Here's the reproducible code

from langchain_core.tools import tool
from langchain_google_genai import ChatGoogleGenerativeAI
from langchain_openai import ChatOpenAI

@tool
def check_weather(location: str) -> float:
    '''Return the weather forecast for the specified location.'''
    return f"It's always sunny in {location}"

@tool
def check_live_traffic(location: str) -> float:
    '''Return the live traffic for the specified location.'''
    return f"The traffic is standstill in {location}"

@tool
def check_tennis_score(player: str) -> float:
    '''Return the latest player's tennis score.'''
    return f"{player} is currently winning 6-0"

tools = [check_weather, check_live_traffic, check_tennis_score]

# model = ChatOpenAI(
#     model="gpt-4o",
#     streaming=True
#     )
model = ChatGoogleGenerativeAI(
    model="gemini-1.5-flash",
    )

model = model.bind_tools(tools)

input = "show me all tools available to you"

result = model.invoke(input)
print(result.content)

here ChatOpenAi response:

I have access to the following tools:

1. **Check Weather**: Provides the weather forecast for a specified location.
2. **Check Live Traffic**: Provides live traffic updates for a specified location.
3. **Check Tennis Score**: Provides the latest tennis score for a specified player.

If you need information or assistance using any of these tools, please let me know!

here is ChatGoogleGenerativeAI response

I have access to the following tools:

* **`default_api`**: This API provides a function to check the weather for a given location.

I can use these tools to answer your questions and complete your requests. For example, I can tell you the current weather in London by using the `check_weather` function from the `default_api`.

You can ask questions about 2nd and 3rd tool like "what is traffic condition in LA?", "what is tennis score for rafael?". Gemini will say "I don't have access to live event" etc etc, while ChatOpenAi called the tools properly as expected.

s99100532 commented 1 month ago

Fyi, the change below yield the desired ouput with model gemini-1.5-flash. (functions is depcreated in openai) Change

model = model.bind_tools(tools)

To

model = model.bind(
    functions=tools,
) 

image

I cannot confirm whether it is a bug or misconconception of usage yet (related code)

Update: https://www.googlecloudcommunity.com/gc/AI-ML/Creating-a-Generative-Model-with-multiple-tools-with-vertexai/m-p/750866 Assuming I understand correctly, u needs to use functions if u have multiple actions