Invalid JSON payload received (Google Provider).

[mishushakov]

Description

When using streamObject, I'm getting the following error with the Google AI provider:

{
    code: 400,
    message: `Invalid JSON payload received. Unknown name "type" at 'generation_config.response_schema.properties[7].value': Proto field is not repeating, cannot start list.`,
    status: 'INVALID_ARGUMENT'
}

Code example

No response

Additional context

using:

"@ai-sdk/google": "^0.0.41"

[mishushakov]

Might be related? #2564

[lgrammel]

The basic stream object example works for Google: https://github.com/vercel/ai/blob/main/examples/ai-core/src/stream-object/google.ts

I suspect it has something to do with the specifics of the schema. Can you share the schema that you are using, or try to narrow it down to the part of the schema that causes the failure?

[sullyo]

Running into this issue as well.

It involves deeply nested zod schemas, which do work on anthropic and openai but on on gemini (broken on both vertex and google studio)

Specifically it breaks when you have a union with zod:

const personSchema = z.object({
  name: z.string(),
  age: z.number(),
  contact: z.union([
    z.object({
      type: z.literal("email"),
      value: z.string(),
    }),
    z.object({
      type: z.literal("phone"),
      value: z.string(),
    }),
  ]),
  occupation: z.union([
    z.object({
      type: z.literal("employed"),
      company: z.string(),
      position: z.string(),
    }),
    z.object({
      type: z.literal("student"),
      school: z.string(),
      grade: z.number(),
    }),
    z.object({
      type: z.literal("unemployed"),
    }),
  ]),
});

Edit: Upon further analysis the issue isn't with the unions, its the type: z.literal("employed")

[lgrammel]

Caused by differences between JSON Schema 7 (used by AI SDK) and subset of OpenAPI Schema 3.0 that is used by Google GenAI & Vertex.

Google does not support anyOf, oneOf, allOf.

[lgrammel]

With a full conversion to OpenAPI schema, I'm running into issues with the Google provider: https://github.com/vercel/ai/pull/2760

Need to investigate if they support anyOf etc:

      message: '* GenerateContentRequest.generation_config.response_schema.properties[occupation].type: must be specified\n' +
        '* GenerateContentRequest.generation_config.response_schema.properties[contact].type: must be specified\n',

[lgrammel]

Unless I'm missing something, Google does not support anyOf etc:

From Vertex:

/**
 * Contains the list of OpenAPI data types
 * as defined by https://swagger.io/docs/specification/data-models/data-types/
 */
export declare enum FunctionDeclarationSchemaType {
    /** String type. */
    STRING = "STRING",
    /** Number type. */
    NUMBER = "NUMBER",
    /** Integer type. */
    INTEGER = "INTEGER",
    /** Boolean type. */
    BOOLEAN = "BOOLEAN",
    /** Array type. */
    ARRAY = "ARRAY",
    /** Object type. */
    OBJECT = "OBJECT"
}
/**
 * Schema for parameters passed to {@link FunctionDeclaration.parameters}.
 */
export interface FunctionDeclarationSchema {
    /** The type of the parameter. */
    type: FunctionDeclarationSchemaType;
    /** The format of the parameter. */
    properties: {
        [k: string]: FunctionDeclarationSchemaProperty;
    };
    /** Optional. Description of the parameter. */
    description?: string;
    /** Optional. Array of required parameters. */
    required?: string[];
}
/**
 * Schema is used to define the format of input/output data.
 * Represents a select subset of an OpenAPI 3.0 schema object.
 * More fields may be added in the future as needed.
 */
export interface FunctionDeclarationSchemaProperty {
    /**
     * Optional. The type of the property. {@link
     * FunctionDeclarationSchemaType}.
     */
    type?: FunctionDeclarationSchemaType;
    /** Optional. The format of the property. */
    format?: string;
    /** Optional. The description of the property. */
    description?: string;
    /** Optional. Whether the property is nullable. */
    nullable?: boolean;
    /** Optional. The items of the property. {@link FunctionDeclarationSchema} */
    items?: FunctionDeclarationSchema;
    /** Optional. The enum of the property. */
    enum?: string[];
    /** Optional. Map of {@link FunctionDeclarationSchema}. */
    properties?: {
        [k: string]: FunctionDeclarationSchema;
    };
    /** Optional. Array of required property. */
    required?: string[];
    /** Optional. The example of the property. */
    example?: unknown;
}

[lgrammel]

I'll need to introduce a flag on the Gemini provider that distinguishes grammar-guided generation from loose generation.

[sullyo]

What would be a good interim solution for that sort of zod schema - or is it better to wait?

[mishushakov]

Here's my schema, if it helps:

export const artifactSchema = z.object({
  commentary: z.string().describe('Describe what you just did and the process of generating the artifact.'),
  template: z.string().describe('Name of the template used to generate the artifact.'),
  title: z.string().describe('Short title of the artifact. Max 5 characters.'),
  description: z.string().describe('Short description of the artifact. Max 10 characters.'),
  additional_dependencies: z.array(z.string()).describe('Additional dependencies required by the artifact. Do not include dependencies that are already included in the template.'),
  has_additional_dependencies: z.boolean().describe('Detect if additional dependencies that are not included in the template are required by the artifact.'),
  install_dependencies_command: z.string().describe('Command to install additional dependencies required by the artifact.'),
  port: z.number().nullable().describe('Port number used by the resulted artifact. Null when no ports are exposed.'),
  code: z.string().describe('Code generated by the artifact.'),
  file_path: z.string().describe('Relative path to the file, including the file name.'),
})

it's pretty flat as you see

[mishushakov]

The field that is causing the error:

port: z.number().nullable().describe('Port number used by the resulted artifact. Null when no ports are exposed.'),

When it is non-nullable then it is working. Any tips?

[lgrammel]

Addressed in the latest version of the google provider, see also https://sdk.vercel.ai/providers/ai-sdk-providers/google-generative-ai#troubleshooting-schema-limitations

[mishushakov]

Works for me 🥳