Error "Unknown / unsupported author: tool" when using Gemini (@langchain/google-genai) with AgentExecutor

s97712 commented 2 months ago

Checked other resources

[X] I added a very descriptive title to this issue.
[X] I searched the LangChain.js documentation with the integrated search.
[X] I used the GitHub search to find a similar question and didn't find it.
[X] I am sure that this is a bug in LangChain.js rather than my code.
[X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

Follow this guide: https://js.langchain.com/v0.2/docs/integrations/chat/google_vertex_ai Replace '@langchain/google-vertexai' with '@langchain/google-genai' and change the relevant code

Error Message and Stack Trace (if applicable)

Error: Unknown / unsupported author: tool
    at convertAuthorToRole (common.js:45:19)
    at messages.reduce.content (common.js:113:22)
    at Array.reduce (<anonymous>)
    at convertBaseMessagesToContent (common.js:105:21)
    at ChatGoogleGenerativeAI._generate (chat_models.js:227:102)
    at eval (chat_models.js:182:96)
    at Array.map (<anonymous>)
    at ChatGoogleGenerativeAI._generateUncached (chat_models.js:182:67)
    at async ChatGoogleGenerativeAI.invoke (chat_models.js:66:24)
    at async RunnableSequence.invoke (base.js:1154:33)
    at async RunnableMultiActionAgent.plan (agent.js:289:22)
    at async AgentExecutor._call (executor.js:431:26)
    at async AgentExecutor.invoke (base.js:63:28)
    at async runWorkflow (Workflow.tsx:41:10)
    at async onClick (page.tsx:18:26)

Description

I try to use gemini with AgentExecutor, I expect it to work like openai.

System Info

windows

dosubot[bot] commented 2 months ago

Hey there, @s97712! I'm here to assist you with any issues you have. Let's get this sorted out for you.

To resolve the "Unknown / unsupported author: tool" error when using Gemini with AgentExecutor in LangChain.js, ensure that the tools you are using are correctly defined and included in the tools array passed to the AgentExecutor. The error typically occurs when the tool specified in the agent's action is not found in the tools array or if there is a mix of structured tools with Gemini tools.

Here is an example of how to correctly define and use tools with AgentExecutor:

import { AgentExecutor, ExceptionTool } from 'path-to-langchain';
import { SerpAPI, Calculator } from 'path-to-tools';

// Define your tools
const tools = [new SerpAPI(), new Calculator()];

// Create an instance of AgentExecutor with the defined tools
const executor = AgentExecutor.fromAgentAndTools({
  agent: async () => loadAgentFromLangchainHub(),
  tools: tools,
  returnIntermediateSteps: true,
});

// Invoke the executor with an input
const result = await executor.invoke({
  input: `Who is Olivia Wilde's boyfriend? What is his current age raised to the 0.23 power?`,
});

console.log(result);

Ensure that the tool names used in the agent's actions match the names of the tools defined in the tools array. Additionally, make sure that you are not mixing structured tools with Gemini tools, as this can cause conflicts. If a tool is not found, the ExceptionTool can be used to handle such cases gracefully ^[1]^[2].

To continue talking to Dosu, mention @dosu.

s97712 commented 2 months ago

function convertAuthorToRole(author) {
    switch (author) {
        /**
         *  Note: Gemini currently is not supporting system messages
         *  we will convert them to human messages and merge with following
         * */
    case "ai":
    case "model":
        // getMessageAuthor returns message.name. code ex.: return message.name ?? type;
        return "model";
    case "system":
    case "human":
        return "user";
    default:
        throw new Error(`Unknown / unsupported author: ${author}`);
    }
}

The error appears here, it looks like langchain doesn't support gemini's ToolMessage message. But I found this on Flowise, Flowise seems to be able to handle ToolMessage messages. https://github.com/FlowiseAI/Flowise/blob/371e632a2c237e208c7cd335aab6912ac5baa8ce/packages/components/nodes/chatmodels/ChatGoogleGenerativeAI/FlowiseChatGoogleGenerativeAI.ts#L323

afirstenberg commented 2 months ago

The "@langchain/google-genai" package is very different than "@langchain/google-vertexai" package. You can't substitute one for the other.

If you want to use the AI Studio API platform instead of the Vertex AI platform, use the "@langchain/googe-gauth" (or "@langchain/google-webauth") package with an AI Studio API Key.

See https://medium.com/@afirstenberg/langchain-js-and-gemini-getting-started-fa602a45bee4 for more about the auth for these packages.

jacoblee93 commented 2 months ago

We definitely need to better document that ^

In the meantime, @bracesproul our standard tests should be catching this?

bracesproul commented 2 months ago

@s97712 what version of @langchain/google-genai are you running? We pushed a new release yesterday with what should contain a fix for this in version 0.0.18

s97712 commented 2 months ago

@bracesproul I tried the latest version, and although there are no errors, it still doesn't work as expected.

This is the last API request. It keeps calling the tools but doesn't seem to convert ToolMessage correctly.

// POST https://generativelanguage.googleapis.com/v1beta/models/gemini-pro:generateContent

{
    "generationConfig": {
        "candidateCount": 1,
        "stopSequences": []
    },
    "safetySettings": [],
    "tools": [
        {
            "functionDeclarations": [
                {
                    "name": "get_current_weather",
                    "description": "Get the current weather in a given location",
                    "parameters": {
                        "type": "object",
                        "properties": {
                            "location": {
                                "type": "string",
                                "description": "The city and state, e.g. San Francisco, CA"
                            }
                        },
                        "required": [
                            "location"
                        ]
                    }
                }
            ]
        }
    ],
    "contents": [
        {
            "role": "user",
            "parts": [
                {
                    "text": "You are a helpful assistant"
                },
                {
                    "text": "New York weather"
                }
            ]
        },
        {
            "role": "model",
            "parts": [
                {
                    "text": ""
                }
            ]
        },
        {
            "role": "user",
            "parts": [
                {
                    "text": "28 °C"
                }
            ]
        },
        {
            "role": "model",
            "parts": [
                {
                    "text": ""
                }
            ]
        },
        {
            "role": "user",
            "parts": [
                {
                    "text": "28 °C"
                }
            ]
        },
        {
            "role": "model",
            "parts": [
                {
                    "text": ""
                }
            ]
        },
        {
            "role": "user",
            "parts": [
                {
                    "text": "28 °C"
                }
            ]
        },
        {
            "role": "model",
            "parts": [
                {
                    "text": ""
                }
            ]
        },
        {
            "role": "user",
            "parts": [
                {
                    "text": "28 °C"
                }
            ]
        }
    ]
}

bracesproul commented 2 months ago

hey @s97712 I believe I've managed to find a fix here.

And when I test with the Vertex agent example (but with ChatGoogleGenerativeAI it works well.

I'll get this merged and released asap.

langchain-ai / langchainjs