ChatGoogleGenerativeAI:TypeError: Parameter to MergeFrom() must be instance of same class: expected google.ai.generativelanguage.v1beta.FunctionDeclaration got FunctionDeclaration. When trying to bind @tools

AkashBais commented 1 week ago

Checked other resources

[X] I added a very descriptive title to this issue.
[X] I searched the LangChain documentation with the integrated search.
[X] I used the GitHub search to find a similar question and didn't find it.
[X] I am sure that this is a bug in LangChain rather than my code.
[X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

I am trying to use ChatGoogleGenerativeAI for tool calling and Agentic Ai implimentation

from langchain_google_genai import ChatGoogleGenerativeAI
model_name = "gemini-1.5-flash-001" #"gemini-1.5-pro-latest"
llm = ChatGoogleGenerativeAI(model=model_name,
                              google_api_key=os.getenv('GOOGLE_API_KEY_2'),
                              max_output_tokens = 1024,
                              temperature = 0,
                              verbose=False,
                              )

llm_with_tools = llm.bind(functions = [search_web, split_documents ] )
responce = llm_with_tools.invoke("Search for LangChain on DuckDuckGo")

Below are the tool definations

@tool(parse_docstring=True)
def split_documents(
    chunk_size: int,
    knowledge_base: Annotated[Union[List[LangchainDocument], LangchainDocument], InjectedToolArg],
    chunk_overlap: Optional[int] = None,
    tokenizer_name: Annotated[Optional[str], InjectedToolArg] = config.EMBEDDING_MODEL_NAME
) -> List[LangchainDocument]:

    """
    This method is an implimentation of Text splitter that uses HuggingFace tokenizer to count length.
    This method is to be called for chuncking a Langchain Document/List of Langchain Documents.
    Returns a list of chuncked LangChain Document(s)

    Args:
      chunk_size: "Size of Chunks to be created, in number of tokens.Depends on the Context window length of the Embedding model"
      knowledge_base: "List of Langchain Document(s) to process. To be passed at run time
      chunk_overlap: "Size of overlap between Chunks, in number of tokens"
      tokenizer_name: "Name of the tokanizer model to be used for tokanization before chunking the Langchain Document(s) in knowledge_base"
    """
   # Tool Code

@tool(parse_docstring=True)
def search_web(
    query:str,
    engine:Optional[str]="Google",
    num_results:Optional[int]=5,
    truncate_threshold:Optional[int]=None,
) -> List[LangchainDocument]:

    """
    Performs web search for the passed query using the desired search engine's API.
    This function will then use web scraping to get page content and return them as a list of LangChain documents.

    Args:
      query:"Query to perform Web search for"
      engine: "The search engine to use for the web search"
      num_results: "The number of search results to return from web search"
      truncate_threshold: "Threshold in number of characters to truncate each web pages' content"
    """

   # Tool Code

Error Message and Stack Trace (if applicable)

WARNING:langchain_google_genai._function_utils:Key 'title' is not supported in schema, ignoring
WARNING:langchain_google_genai._function_utils:Key 'title' is not supported in schema, ignoring
WARNING:langchain_google_genai._function_utils:Key 'title' is not supported in schema, ignoring
WARNING:langchain_google_genai._function_utils:Key 'default' is not supported in schema, ignoring
WARNING:langchain_google_genai._function_utils:Key 'title' is not supported in schema, ignoring
WARNING:langchain_google_genai._function_utils:Key 'default' is not supported in schema, ignoring
WARNING:langchain_google_genai._function_utils:Key 'title' is not supported in schema, ignoring

TypeError Traceback (most recent call last) in <cell line: 1>() ----> 1 responce = llm_with_tools.invoke("Search for LangChain on DuckDuckGo")

8 frames /usr/local/lib/python3.10/dist-packages/langchain_core/runnables/base.py in invoke(self, input, config, kwargs) 5090 kwargs: Optional[Any], 5091 ) -> Output: -> 5092 return self.bound.invoke( 5093 input, 5094 self._merge_configs(config),

/usr/local/lib/python3.10/dist-packages/langchain_core/language_models/chat_models.py in invoke(self, input, config, stop, **kwargs) 275 return cast( 276 ChatGeneration, --> 277 self.generate_prompt( 278 [self._convert_input(input)], 279 stop=stop,

/usr/local/lib/python3.10/dist-packages/langchain_core/language_models/chat_models.py in generate_prompt(self, prompts, stop, callbacks, kwargs) 775 ) -> LLMResult: 776 prompt_messages = [p.to_messages() for p in prompts] --> 777 return self.generate(prompt_messages, stop=stop, callbacks=callbacks, kwargs) 778 779 async def agenerate_prompt(

/usr/local/lib/python3.10/dist-packages/langchain_core/language_models/chat_models.py in generate(self, messages, stop, callbacks, tags, metadata, run_name, run_id, **kwargs) 632 if run_managers: 633 run_managers[i].on_llm_error(e, response=LLMResult(generations=[])) --> 634 raise e 635 flattened_outputs = [ 636 LLMResult(generations=[res.generations], llm_output=res.llm_output) # type: ignore[list-item]

/usr/local/lib/python3.10/dist-packages/langchain_core/language_models/chat_models.py in generate(self, messages, stop, callbacks, tags, metadata, run_name, run_id, **kwargs) 622 try: 623 results.append( --> 624 self._generate_with_cache( 625 m, 626 stop=stop,

/usr/local/lib/python3.10/dist-packages/langchain_core/language_models/chat_models.py in _generate_with_cache(self, messages, stop, run_manager, kwargs) 844 else: 845 if inspect.signature(self._generate).parameters.get("run_manager"): --> 846 result = self._generate( 847 messages, stop=stop, run_manager=run_manager, kwargs 848 )

/usr/local/lib/python3.10/dist-packages/langchain_google_genai/chat_models.py in _generate(self, messages, stop, run_manager, tools, functions, safety_settings, tool_config, generation_config, kwargs) 964 kwargs: Any, 965 ) -> ChatResult: --> 966 request = self._prepare_request( 967 messages, 968 stop=stop,

/usr/local/lib/python3.10/dist-packages/langchain_google_genai/chat_models.py in _prepare_request(self, messages, stop, tools, functions, safety_settings, tool_config, generation_config) 1128 formatted_tools = [convert_to_genai_function_declarations(tools)] 1129 elif functions: -> 1130 formatted_tools = [convert_to_genai_function_declarations(functions)] 1131 1132 system_instruction, history = _parse_chat_history(

/usr/local/lib/python3.10/dist-packages/langchain_google_genai/_function_utils.py in convert_to_genai_function_declarations(tools) 177 else: 178 fd = _format_to_gapic_function_declaration(tool) --> 179 gapic_tool.function_declarations.append(fd) 180 return gapic_tool 181

TypeError: Parameter to MergeFrom() must be instance of same class: expected google.ai.generativelanguage.v1beta.FunctionDeclaration got FunctionDeclaration.

Description

I am trying to use ChatGoogleGenerativeAI for tool calling and Agentic Ai implimentation. With responce = llm_with_tools.invoke("Search for LangChain on DuckDuckGo") I expect it to call search_web with appropriate arguments. But instead it give me the following error.

System Info

System Information

OS: Linux OS Version: #1 SMP PREEMPT_DYNAMIC Thu Jun 27 21:05:47 UTC 2024 Python Version: 3.10.12 (main, Jul 29 2024, 16:56:48) [GCC 11.4.0]

Package Information

langchain_core: 0.2.37 langchain: 0.2.15 langchain_community: 0.2.15 langsmith: 0.1.108 langchain_google_genai: 1.0.10 langchain_text_splitters: 0.2.2

Optional packages not installed

langgraph langserve

Other Dependencies

aiohttp: 3.10.5 async-timeout: 4.0.3 dataclasses-json: 0.6.7 google-generativeai: 0.7.2 httpx: 0.27.2 jsonpatch: 1.33 numpy: 1.26.4 orjson: 3.10.7 packaging: 24.1 pillow: 10.4.0 pydantic: 2.8.2 PyYAML: 6.0.2 requests: 2.32.3 SQLAlchemy: 2.0.32 tenacity: 8.5.0 typing-extensions: 4.12.2

thanhtan1105 commented 1 week ago

I got the same issue? Our code did not change also core library :(

AkashBais commented 1 week ago

Hi @eyurtsev, Hope you are doing well. Can you help us out with this issue . We’d really appreciate your help! Thanks

thanhtan1105 commented 1 week ago

Hi @AkashBais I have a idea, you should remove all environment and install again. My problem has solved for action. I am using conda environment. I have remove it and create env with same name. Luckily, it work for me. It take me many time

AkashBais commented 1 week ago

@thanhtan1105 I tried it but it dosent do the trick for me. Will need a more permanent soution. For you is it just re creating the env or you made some other changes too.

thanhtan1105 commented 1 week ago

Hi @AkashBais you can follow my library version below with requirement.txt file. I did not make any change, destroy env and install again. You can try it

langchain==0.2.15 ollama==0.3.2 huggingface-hub==0.24.6 opensearch-py==2.7.1 langchain-community unstructured==0.15.9 markdown==3.7 nltk==3.9.1 llama-index-vector-stores-opensearch==0.1.14 llama-index-embeddings-ollama==0.2.0 llama-index-embeddings-langchain==0.1.2 llama-index-llms-llama-cpp==0.1.4 llama-index-core==0.10.68 llama-index-llms-langchain==0.3.0 langchain-openai==0.1.23 langchainhub==0.1.21 google-generativeai==0.7.1 grpcio==1.66.1 grpcio-tools==1.62.3 langchain-google-genai==1.0.10 fastapi==0.112.2 uvicorn==0.30.6 langchain-core==0.2.38 google-cloud-aiplatform==1.64.0 python-dotenv==1.0.1 google-cloud-texttospeech==2.17.2

AkashBais commented 1 week ago

HI @thanhtan1105. I am using langchain == 0.2.15 and langchain_google_genai == 1.0.10. I guess if it's a compatablity issue it should be between these packages, but they are the same as in your discription. Also a quick question if I may. How are you defining the Tools for routing and function call. Maybe that's where the issue is with my code.

thanhtan1105 commented 1 week ago

Hi @AkashBais I just create dummy tools without any action. I can share little bit in the image to you. Very simple llm.bind_tools functions

thanhtan1105 commented 1 week ago

I am consider move all code base to llamaindex framework. I think it more stable than langchain. I hope that.

AkashBais commented 1 week ago

@thanhtan1105 Thanks for the help, but I am still stuck. I tried ChatVertexAI and ChatGoogleGenerativeAI from langchain. Both have the same issue. Let's see if there is a resolution on this opened issue.

AkashBais commented 1 week ago

Closing this issue.Raising a new issue with more detailed description.

langchain-ai / langchain