Tool usage in ChatGoogleGenerativeAI is totally broken and so are the tests for it

[Pupix]

Checked other resources

• [X] I added a very descriptive title to this issue.
• [X] I searched the LangChain.js documentation with the integrated search.
• [X] I used the GitHub search to find a similar question and didn't find it.
• [X] I am sure that this is a bug in LangChain.js rather than my code.
• [X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

Any of the tutorials that use tools will do.

Error Message and Stack Trace (if applicable)

No response

Description

While doing the tutorial about the weather tool, I noticed that the AI would keep responding as if I was the one telling it the weather and not the other way around.

The code seems to early exit if there's a BaseMessage with string contents, without checking if it's from a tool or not.

It can be seen here https://github.com/langchain-ai/langchainjs/blob/eb9ce8d05ab5bc5fc07b24c33e651943149c07fe/libs/langchain-google-genai/src/utils/common.ts#L86-L92

There are tool checks a few lines down, but it will never reach if the tool just responds with 'It's 60 degrees.'

More than that, looking at the message formatting, the tool responses must be objects and not strings like they are sent here. https://github.com/langchain-ai/langchainjs/blob/eb9ce8d05ab5bc5fc07b24c33e651943149c07fe/libs/langchain-google-genai/src/utils/common.ts#L110-L114

I tried returning objects from the tool, but it stringified them somewhere.

The typedef for FunctionResponse

From: https://github.com/google-gemini/generative-ai-js/blob/cec7440c0d34d29298333d920d4f44f8478d27d5/types/content.ts#L160

The tests are also broken. They run as if the users tells the weather or whatever to the AI instead of properly using functionResponse. https://github.com/langchain-ai/langchainjs/blob/eb9ce8d05ab5bc5fc07b24c33e651943149c07fe/libs/langchain-google-genai/src/tests/chat_models.test.ts#L184-L201

As a workaround I monkeypatched langchain by moving the above mentioned string check at the bottom of the if/else if block

and wrapped response in functionResponse in an object.

Probably not the best way to handle things, but it works.

System Info

Windows Node 22.x.x yarn 1.x.x

[dosubot[bot]]

Hey @Pupix! 👋 I'm here to help you with any bugs, questions, or contributions you have in mind. Let's squash those bugs together!

I found a similar unsolved discussion that might be relevant to your issue: langchain agent -> Exiting LLM run with output: { "generations": [ [ null ] ] } ^[1]. Additionally, there was a closed issue related to providing functionResponses, which suggested ensuring that the values of input variables are strings before invoking methods ^[2].

To address the problem where tool responses are incorrectly processed as user messages, you need to ensure that tool responses are structured as objects using the FunctionResponse interface. The LangChain.js codebase should correctly parse tool responses and create an AIMessage with the appropriate content. Here's a snippet that demonstrates how to handle tool responses correctly:

const chatGenerationContent = chatResult.generations[0].message.content;
if (typeof chatGenerationContent !== "string") {
  throw new Error("OllamaFunctions does not support non-string output.");
}
let parsedChatResult;
try {
  parsedChatResult = JSON.parse(chatGenerationContent);
} catch (e) {
  throw new Error(
    `"${this.llm.model}" did not respond with valid JSON. Please try again.`
  );
}
const calledToolName = parsedChatResult.tool;
const calledToolArguments = parsedChatResult.tool_input;
const calledTool = functions.find((fn) => fn.name === calledToolName);
if (calledTool === undefined) {
  throw new Error(
    `Failed to parse a function call from ${this.llm.model} output: ${chatGenerationContent}`
  );
}
if (calledTool.name === this.defaultResponseFunction.name) {
  return {
    generations: [
      {
        message: new AIMessage({
          content: calledToolArguments.response,
        }),
        text: calledToolArguments.response,
      },
    ],
  };
}

const responseMessageWithFunctions = new AIMessage({
  content: "",
  additional_kwargs: {
    function_call: {
      name: calledToolName,
      arguments: calledToolArguments
        ? JSON.stringify(calledToolArguments)
        : "",
    },
  },
});

return {
  generations: [{ message: responseMessageWithFunctions, text: "" }],
};

This code ensures that tool responses are parsed and structured correctly, preventing them from being misinterpreted as user messages ^[3][4].

To continue talking to Dosu, mention @dosu.