[bugifx] ShellTool not work for gemini-1.5 Search because of TYPE_ENUM[v["type"]] returns KeyError: 'type'

[nobu007]

Example Code

from dotenv import load_dotenv
from langchain import hub
from langchain.agents import AgentExecutor, create_tool_calling_agent
from langchain_anthropic import ChatAnthropic
from langchain_community.tools.shell.tool import ShellInput, ShellTool
from langchain_core.utils.function_calling import format_tool_to_openai_function
from langchain_google_genai import ChatGoogleGenerativeAI

load_dotenv(verbose=True, override=False)

is_anthropic = False
if is_anthropic:
    model_anthropic = "claude-3-haiku-20240307"
    llm_chat = ChatAnthropic(
        model_name=model_anthropic,
    )
else:
    model_google = "gemini-1.5-pro-latest"
    model_google = "gemini-1.5-flash-latest"
    # model_google = "gemini-1.0-pro" # not accept instruction message.
    llm_chat = ChatGoogleGenerativeAI(
        model=model_google,
    )

shell_tool = ShellTool()
tools = [shell_tool]
llm_chat_with_tools = llm_chat.bind_tools(tools)  # not impl yet

question = "please run pwd command."
agent_scratchpad = []

prompt = hub.pull("hwchase17/openai-functions-agent")
agent = create_tool_calling_agent(llm_chat, tools, prompt)
agent_executor = AgentExecutor(
    agent=agent,
    tools=tools,
    allowed_tools=[shell_tool.name],
)

input_message = {"input": question, "agent_scratchpad": agent_scratchpad}
result = agent_executor.invoke(input_message)

print("result=", result)

Error Message and Stack Trace

Traceback (most recent call last):
  File "/home/jinno/git/drill/gamebook/codeinterpreter_api_agent/../test/test_shell_tool.py", line 38, in <module>
    agent = create_tool_calling_agent(llm_chat, tools, prompt)
  File "/home/jinno/.local/share/virtualenvs/gamebook-7W9hF6PY/lib/python3.10/site-packages/langchain/agents/tool_calling_agent/base.py", line 88, in create_tool_calling_agent
    llm_with_tools = llm.bind_tools(tools)
  File "/home/jinno/.local/share/virtualenvs/gamebook-7W9hF6PY/lib/python3.10/site-packages/langchain_google_genai/chat_models.py", line 906, in bind_tools
    genai_tools = [tool_to_dict(convert_to_genai_function_declarations(tools))]
  File "/home/jinno/.local/share/virtualenvs/gamebook-7W9hF6PY/lib/python3.10/site-packages/langchain_google_genai/_function_utils.py", line 65, in convert_to_genai_function_declarations
    return convert_to_genai_function_declarations({"function_declarations": tool})
  File "/home/jinno/.local/share/virtualenvs/gamebook-7W9hF6PY/lib/python3.10/site-packages/langchain_google_genai/_function_utils.py", line 68, in convert_to_genai_function_declarations
    function_declarations=[
  File "/home/jinno/.local/share/virtualenvs/gamebook-7W9hF6PY/lib/python3.10/site-packages/langchain_google_genai/_function_utils.py", line 69, in <listcomp>
    _convert_to_genai_function(fc) for fc in tool["function_declarations"]
  File "/home/jinno/.local/share/virtualenvs/gamebook-7W9hF6PY/lib/python3.10/site-packages/langchain_google_genai/_function_utils.py", line 105, in _convert_to_genai_function
    return _convert_tool_to_genai_function(fc)
  File "/home/jinno/.local/share/virtualenvs/gamebook-7W9hF6PY/lib/python3.10/site-packages/langchain_google_genai/_function_utils.py", line 137, in _convert_tool_to_genai_function
    "properties": {
  File "/home/jinno/.local/share/virtualenvs/gamebook-7W9hF6PY/lib/python3.10/site-packages/langchain_google_genai/_function_utils.py", line 139, in <dictcomp>
    "type_": TYPE_ENUM[v["type"]],
KeyError: 'type'
Error in sys.excepthook:
Traceback (most recent call last):
  File "/home/jinno/.local/share/virtualenvs/gamebook-7W9hF6PY/lib/python3.10/site-packages/exceptiongroup/_formatting.py", line 71, in exceptiongroup_excepthook
TypeError: 'NoneType' object is not callable

Description

ChatAnthropic works fine. But ChatGoogleGenerativeAI shows error. It looks be caused by here.

1. TYPE_ENUM[v["type"]] is no care for None case in here.
2. Return type of here is wrong.

System Info

Ubuntu 23.04

langchain==0.2.0 langchain-anthropic==0.1.11 langchain-community==0.0.34 langchain-core==0.2.0 langchain-experimental==0.0.57 langchain-google-genai==1.0.4 langchain-groq==0.1.3 langchain-openai==0.1.6 langchain-text-splitters==0.2.0 langchainhub==0.1.15

[ZhymabekRoman]

@nobu007 Have you solved this problem?

[nobu007]

@ZhymabekRoman I have updated the PR yesterday. It solves the problem in my environment. https://github.com/langchain-ai/langchain-google/pull/233