Responses not adhering to Zod schema with OpenAI models

[manishrc]

Description

I'm not certain if the issue lies in how the Zod schema is translated for OpenAI or with the OpenAI model itself. Some Zod validations are not being adhered to. (It would be nice to also get access to the rawRequest, similar to rawResponse)

For example, the validation z.number.max(60) expects a number not higher than 60, but frequently receives higher values. In this particular case, the prompt was explicitly set to bias towards larger numbers to make the issue easier to reproduce, but the problem occurred without such prompts.

Code example

import { openai } from '@ai-sdk/openai';
import { generateObject } from 'ai';
import { z } from 'zod';

const { rawResponse, ...result } = await generateObject({
  model: openai('gpt-4'),
  schema: z
    .object({
      name: z.string(),
      age: z.number().max(60),
    })
    .catch(({ input, error }) => {
      console.error({
        input,
        error: error.errors,
      });
    }),
  prompt: 'Generate a creative user profile of the oldest person ever',
});

process.stdout.write(JSON.stringify(result, null, 2));

Received output:

{
  input: { name: 'Jeanne Calment', age: 122 },
  error: [
    {
      code: 'too_big',
      maximum: 60,
      type: 'number',
      inclusive: true,
      exact: false,
      message: 'Number must be less than or equal to 60',
      path: [Array]
    }
  ]
}
{
  "finishReason": "stop",
  "usage": {
    "promptTokens": 63,
    "completionTokens": 16,
    "totalTokens": 79
  },
  "warnings": []
}

Additional context

No response

[manishrc]

Also, are there any guidelines for the best way to catch validation errors when using streamObject? I tried using fullStream, but the Zod validations do not appear there.

I see that onFinish has an error, but wondering if it's possible to abort the stream when Zod validation fails.

[lgrammel]

@manishrc currently only the final object is validated with the zod schema. You can find tips here: https://sdk.vercel.ai/docs/ai-sdk-core/tools-and-tool-calling#prompt-engineering-with-tools

My recommendation is to use description() in those cases.