Tool using with gemini

[eyurtsev]

This seems to fail 50% with no completions from the server

from langchain_core.messages import AIMessage, HumanMessage, ToolMessage
from langchain_google_vertexai import ChatVertexAI

model = ChatVertexAI(model_name="gemini-pro")

messages = [
    HumanMessage(
        content=(
            "Repeat the given string using the provided tools. "
            "Do not write anything else or provide any explanations. "
            "For example, if the string is 'abc', you must print the "
            "letters 'a', 'b', and 'c' one at a time and in that order. "
            "\n\n type this: 'a'"
        )
    ),
    AIMessage(
        content="",
        additional_kwargs={
            "function_call": {
                "name": "type_letter",
                "arguments": '{"letter": "a"}',
            }
        },
        tool_calls=[
            {
                "name": "type_letter",
                "args": {"letter": "a"},
                "id": "abc123",
            },
        ],
    ),
    ToolMessage(
        name="type_letter",
        content="OK",
        tool_call_id="abc123",
    ),
]

result = list(model.invoke(messages))

[lkuligin]

It looks to be on the model's side, it returns an empty response like

usage_metadata {
  prompt_token_count: 71
  total_token_count: 71
}

We can probably raise a ValueError when there are no candidates in the response, wdyt?

[eyurtsev]

@lkuligin Are we able to rule out the integration? Is it easy to reproduce with the vertex SDK directly? Could it be that we're sending some incorrect data via the integration and the API isn't validating inputs properly and then failing to produce a response?

[lkuligin]

@eyurtsev I've reproduced it with the native SDK only and raised an issue internally.

[ccurme]

@lkuligin I've noticed that in the streaming context, we seem to consistently get an IndexError if we try to invoke a tool in a sequence more than two times. Here is an example:

import random

from langchain.agents import AgentExecutor, create_tool_calling_agent
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_core.tools import tool
from langchain_google_vertexai import ChatVertexAI

instructions = (
    "You are using a translation machine. "
    "You will be given a string. Input the characters one at a "
    "time into the machine. The machine will give you output for "
    "each character. When finished, provide the final output. "
    "For example, if the string is 'abc', you must input the "
    "letters 'a', 'b', and 'c' one at a time and in that order. "
    "The responses may be, for example, 'z', 'x', 'q'. You will then "
    "report the final output as 'zxq'. "
    "\n\n {input}"
)

prompt = ChatPromptTemplate.from_messages(
    [
        ("human", instructions),
        MessagesPlaceholder("agent_scratchpad"),
    ]
)

model = ChatVertexAI(model_name="gemini-1.5-pro-preview-0409")

@tool
def type_letter(letter: str) -> str:
    """Type the given letter."""

    alphabet = "abcdefghijklmnopqrstuvwxyz"
    return alphabet[random.randint(0, 25)]

tools = [type_letter]

agent = create_tool_calling_agent(model, tools, prompt)

executor = AgentExecutor(
    agent=agent,
    tools=tools,
    verbose=True,
)

res = executor.invoke({"input": "type this: 'ba'"})  # works (agent reports output)

res = executor.invoke({"input": "type this: 'bat'"})  # hits IndexError

If this is expected, one option is to return a message at this point, e.g., in https://github.com/langchain-ai/langchain-google/pull/162