langchain-ai / langchainjs

🦜🔗 Build context-aware reasoning applications 🦜🔗
https://js.langchain.com/docs/
MIT License
12.71k stars 2.19k forks source link

Google Gen AI Structured Output Error #6901

Open tonyabracadabra opened 1 month ago

tonyabracadabra commented 1 month ago

Checked other resources

Example Code

import { ChatGoogleGenerativeAI } from "@langchain/google-genai";

import { z, ZodTypeAny } from "zod";

interface GenerateStructuredOutputParams<T extends ZodTypeAny> {
  schema: T;
  request: string;
  temperature?: number;
}

export async function generateStructuredOutput<T extends ZodTypeAny>({
  schema,
  request,
  temperature = 1.5,
}: GenerateStructuredOutputParams<T>): Promise<z.infer<T>> {
  const model = new ChatGoogleGenerativeAI({
    model: "gemini-1.5-flash",
    temperature,
  });

  const structuredLlm = model.withStructuredOutput(schema);
  return await structuredLlm.invoke(request);
}

Error Message and Stack Trace (if applicable)

⨯ node_modules/@langchain/google-genai/dist/output_parsers.js (65:1) @ GoogleGenerativeAIToolsOutputParser.parseResult
 ⨯ Error: No parseable tool calls provided to GoogleGenerativeAIToolsOutputParser.
    at async generateStructuredOutput (./src/server/lib/ai/llm.ts:13:12)
    at async generateSingleScript (./src/server/actions/product/script.ts:27:20)
    at async Promise.all (index 1)
    at async generateProductScripts (./src/server/actions/product/script.ts:43:21)
digest: "672233516"
  63 |         });
  64 |         if (tools[0] === undefined) {
> 65 |             throw new Error("No parseable tool calls provided to GoogleGenerativeAIToolsOutputParser.");
     | ^
  66 |         }
  67 |         const [tool] = tools;
  68 |         const validatedResult = await this._validateResult(tool.args);
 POST /dashboard 500 in 3059ms

Description

When attempting to generate structured output using Google's Generative AI (Gemini) through LangChain, an error occurs: CopyError: No parseable tool calls provided to GoogleGenerativeAIToolsOutputParser. Context

The error occurs when using the ChatGoogleGenerativeAI class from @langchain/google-genai. The code is attempting to use structured output with a Zod schema. The model being used is "gemini-1.5-flash".

System Info

Operating System: Platform: darwin Arch: arm64 Version: Darwin Kernel Version 23.5.0: Wed May 1 20:17:33 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T6031 Available memory (MB): 98304 Available CPU cores: 14 Binaries: Node: 20.11.1 npm: 10.2.4 Yarn: N/A pnpm: 9.6.0 Relevant Packages: next: 15.0.0-canary.160 // There is a newer canary version (15.0.0-canary.173) available, please upgrade! eslint-config-next: 15.0.0-canary.69 react: 19.0.0-rc-778e1ed2-20240926 react-dom: 19.0.0-rc-778e1ed2-20240926 typescript: 5.6.2 Next.js Config: output: N/A

dosubot[bot] commented 1 month ago

I found a similar issue that might be relevant to your problem: No tools_call in message error on ChatVertexAI. This issue was closed, and the solution involved ensuring that the input message and schema were correctly aligned with the expected output from the model [1].

To resolve the "No parseable tool calls provided to GoogleGenerativeAIToolsOutputParser" error, ensure that the generations array passed to the parseResult method contains messages with a tool_calls property that is an array. Here is an example of how the generations array should be structured:

const generations = [
  {
    message: {
      tool_calls: [
        {
          args: {
            // your tool call arguments here
          }
        }
      ]
    }
  }
];

Make sure your input data is correctly formatted to include the necessary tool_calls array [2].

To continue talking to Dosu, mention @dosu.

jacoblee93 commented 1 month ago

Does this persist with larger models? What if you prompt more strictly to include something like "you must always use tools if available"?

Unfortunately tool calling/structured output isn't perfect in all cases - CC @bracesproul to make sure we're doing all we can here.

tonyabracadabra commented 1 month ago

Does this persist with larger models? What if you prompt more strictly to include something like "you must always use tools if available"?

Unfortunately tool calling/structured output isn't perfect in all cases - CC @bracesproul to make sure we're doing all we can here.

I don't understand, I am just using it for structured output without requiring it to use any tools.

jeloi commented 1 month ago

hey there - running into the same issue and I believe it's because withStructuredOutput does not call llm.bind with the tool_choice param that @bracesproul added here: https://github.com/langchain-ai/langchainjs/pull/6195/files.

Also relevant is that it looks like "forced function calling" support with "mode: any" was only recently added for Gemini Flash: https://github.com/google-gemini/generative-ai-js/issues/190. I don't think a change is needed specifically to support flash, but would be great if we could update withStructuredOutput to use forced function calling! Currently withStructuredOutput is unusable at least for me.

I think just accepting a param for tool_choice would work - could have an option that automatically sets tool_choice to the explicitly defined the function name, but since there's only one function setting it to "any" should always work.

BaharChidem commented 3 weeks ago

Hi there! We are a group of 3 students from the University of Toronto and we are very interested in fixing this issue and also adding some tests. We will submit a PR for this issue by end of November.

jacoblee93 commented 3 weeks ago

Yes please! Will assign it to you @BaharChidem.

boehlerlukas commented 4 days ago

We are running into the same issue. If there is anything we can help with, let me know.

tonyabracadabra commented 4 days ago

I have switched to vercel's generateObject, life is so much easier now!

austin-duff-prft commented 2 days ago

Any update on this issue? Thanks

bracesproul commented 2 days ago

@tonyabracadabra could you send me an example zod schema that will trigger this error?

tonyabracadabra commented 1 day ago

@tonyabracadabra could you send me an example zod schema that will trigger this error?

I believe any zod schema would trigger this if no tools were specified somewhere within

BaharChidem commented 4 hours ago

We are continuing to work on the issue and hope to have a PR up by the end of next week. We are currently just finding difficulties with replicating the issue in our Linux VMs, but hope to get this resolved soon. Thanks!

boehlerlukas commented 4 hours ago

@BaharChidem thanks a lot!