google-common [Feature]: Grounding with Google Search and Vertex AI Search

[afirstenberg]

Privileged issue

• [X] I am a LangChain maintainer, or was asked directly by a LangChain maintainer to create an issue here.

Issue Content

See https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/grounding

See https://github.com/langchain-ai/langchain-google/issues/146

[afirstenberg]

Do we have a standard way of defining non-FunctionCall tools? (@jacoblee93)

[jacoblee93]

Do we have a standard way of defining non-FunctionCall tools? (@jacoblee93)

Not at the moment - thinking so far has been to always support the model-specific format and make the .tool_calls format on AIMessage expressive enough to handle the various possibilities.

[afirstenberg]

My thinking has been that there would be a function call version of it (for most models to use), but also a function call that a specific model could check for and then use.

So a Gemini model, for example, might check for a function toGeminiTool() or something and, if it exists, call it and use those results instead.

[williamc99]

Hi @afirstenberg and @jacoblee93, we are students from the University of Toronto and we are interested in contributing to this issue. We noticed that a similar issue: langchain-ai/langchain-google/issues/146 was recently completed, so we were wondering if this issue is still open for contribution. If so, we would like to implement a similar solution based on the existing one from langchain-google. Thanks! 😄

[jacoblee93]

Hey @williamc99! I think this should work if you call .bindTools() with Google's format but haven't verified. Would be a nice one to add if you're willing to take it on!

Sorry for the delayed response

[williamc99]

No worries, my team and I will get started right away!

[afirstenberg]

@williamc99 - Please keep me in the loop on what you and your team are seeing or if you have questions.

When I first looked at this, I had several thoughts around this:

• Google is using Tool Calls to implement both function calling and other tools. So functions are a subset of all tools.
• As such, the ToolCallRaw (as it stands now) type isn't appropriate for other Gemini tools, such as the grounding tools.
  • So my thought would be to rename ToolCallRaw to FunctionCallToolRaw
  • We might then create other type definitions for GoogleRetrievalToolRaw and CodeExecutionToolRaw and others that may come up in the future.
  • All of these tools would need some way to distinguish them from normal function tools that other models use. Tho some of them (such as the Retrieval tool) do have an API that can be used outside of Gemini. But that might be for another day.
  • We need this because they aren't included in the list of functions when we create the JSON.
  • The new ToolCallRaw would then be type ToolCallRaw = FunctionCallToolRaw | GoogleRetrievalToolRaw | CodeExecutionToolRaw
• The GeminiTool type will need to reflect the optional retrieval parameter that is sent via JSON. (And this is where other Google tools such as code execution would also end up being defined.)
• With all of that, you would then modify the structuredToolsToGeminiTools() function so it returns GeminiTool objects that handle the different ypes of tools (functiont ools, retrieval tools, etc) in the format that will be sent in the JSON.

I may have misunderstood something, or things may have changed since I last looked at it, so take a look and we can discuss.

[williamc99]

Thank you, these will be great to keep into consideration. We'll keep you updated and reach out if we have any questions. 😀

[afirstenberg]

@williamc99 - Wanted to followup and see how it's going.

Google just announced Grounding with Google Search as a tool that is available through both the Vertex AI API and AI Studio API. My cursory look says that the JSON is identical for both platforms, so this part should be straightforward.

[williamc99]

@afirstenberg - Thanks for following up, and sorry for the delay!

I took a look at the link and I agree that it should just be a straightforward addition. A question I had was whether we should stick to the default threshold for dynamic retrieval (0.3) or if there was a specific value that we should use?

Also, for full transparency, we just finished with midterm exams and it slowed our progress more than expected. We're currently learning the codebase but we expect to have a lot more time to work on the feature now.

We appreciate your guidance and will keep you updated on our progress!

[afirstenberg]

I think it should be easy enough to make that a parameter to the tool and to default to not providing a value (and thus using the default that Google is using). All of that is the easy part! 😅

The more difficult part will be the changes that I outlined above.

[renathossain]

Hello @afirstenberg and @jacoblee93, I am from the same team as @williamc99. We deeply apologize for the delay, but we have good news! We’ve implemented grounding with Google Search and Vertex AI Search. You can find our code here: GitHub Repository.

To use this feature, simply call .bindTools([searchRetrievalTool]) on the model and pass in the appropriate searchRetrievalTool object, as defined in the Google Cloud documentation on grounding, depending on the type of grounding you want to use.

Here are the implementation details:

• We’ve added the necessary googleSearchRetrieval and retrieval parameters to GeminiTool.
• convertToGeminiTools and formatTools previously supported only one tool type (functionDeclarations), but we’ve refactored them to handle these two new tools.
• The structuredToolsToGeminiTools function has been removed; its logic is now consolidated within formatTools.

We've also added test cases and documentation changes.

We have initiated a pull request. Please review our code and let us know your feedback.