AI_TypeValidationError: Type validation failed, even though the response contains the correct data

[davidbarker]

Describe the bug AI_TypeValidationError: Type validation failed, even though the response contains the correct data.

To Reproduce I sent a request, with an image, to LLaVa via the ollama-ai-provider using Vercel AI's generateObject(), and the response in the Network tab is:

{
    "model": "llava",
    "created_at": "2024-08-24T19:29:03.432539Z",
    "message": {
        "role": "assistant",
        "content": "{\"type\":\"object\",\"properties\":{\"detailed_description\":\"The image features a 3D animated character with a friendly and approachable appearance. The character has short brown hair, glasses, and is wearing a black and white striped shirt. They are standing against a dark background, which contrasts with their light-colored outfit. The character's pose suggests they are looking directly at the viewer, giving an impression of engagement and interaction.\"},\"required\":[\"detailed_description\"],\"additionalProperties\":false,\"$schema\":\"http://json-schema.org/draft-07/schema#\"} "
    },
    "done_reason": "stop",
    "done": true,
    "total_duration": 15464915666,
    "load_duration": 57410250,
    "prompt_eval_count": 1,
    "prompt_eval_duration": 8002691000,
    "eval_count": 129,
    "eval_duration": 7383628000
}

However, this leads to the error:

{
  "name": "AI_TypeValidationError",
  "message": "Type validation failed: Value: {\"type\":\"object\",\"properties\":{\"detailed_description\":\"The image features a 3D animated character with a friendly and approachable appearance. The character has short brown hair, glasses, and is wearing a black and white striped shirt. They are standing against a dark background, which contrasts with their light-colored outfit. The character's pose suggests they are looking directly at the viewer, giving an impression of engagement and interaction.\"},\"required\":[\"detailed_description\"],\"additionalProperties\":false,\"$schema\":\"http://json-schema.org/draft-07/schema#\"}.\nError message: [\n  {\n    \"code\": \"invalid_type\",\n    \"expected\": \"string\",\n    \"received\": \"undefined\",\n    \"path\": [\n      \"detailed_description\"\n    ],\n    \"message\": \"Required\"\n  }\n]",
  "cause": {
    "issues": [
      {
        "code": "invalid_type",
        "expected": "string",
        "received": "undefined",
        "path": [
          "detailed_description"
        ],
        "message": "Required"
      }
    ],
    "name": "ZodError"
  },
  "stack": "...",
  "value": {
    "type": "object",
    "properties": {
      "detailed_description": "The image features a 3D animated character with a friendly and approachable appearance. The character has short brown hair, glasses, and is wearing a black and white striped shirt. They are standing against a dark background, which contrasts with their light-colored outfit. The character's pose suggests they are looking directly at the viewer, giving an impression of engagement and interaction."
    },
    "required": [
      "detailed_description"
    ],
    "additionalProperties": false,
    "$schema": "http://json-schema.org/draft-07/schema#"
  }
}

Specifically, I get the error that detailed_description is undefined instead of string.

Expected behavior Looking at the response, the type checking should pass — the detailed_description property is there and is a string.

Additional context Is this likely to be a Vercel AI issue, an ollama-ai-provider issue, or a Zod issue?

[davidbarker]

I just tried requesting tags (via an array) instead, and it seems like LLaVa correctly provides them but I still get an error that it's undefined.

Response

{
    "model": "llava",
    "created_at": "2024-08-24T19:36:05.475219Z",
    "message": {
        "role": "assistant",
        "content": "{\"type\":\"object\",\"properties\":{\"tags\":{\"type\":\"array\",\"items\":[\"glasses\",\"eyeglasses\",\"nerd glasses\",\"geek\",\"hipster\"]}},\"required\":[\"tags\"],\"additionalProperties\":false,\"$schema\":\"http://json-schema.org/draft-07/schema#\"} "
    },
    "done_reason": "stop",
    "done": true,
    "total_duration": 11225521000,
    "load_duration": 21484959,
    "prompt_eval_count": 1,
    "prompt_eval_duration": 7197490000,
    "eval_count": 71,
    "eval_duration": 3979676000
}

Error:

{
  "error": {
    "name": "AI_TypeValidationError",
    "message": "Type validation failed: Value: {\"type\":\"object\",\"properties\":{\"tags\":{\"type\":\"array\",\"items\":[\"glasses\",\"eyeglasses\",\"nerd glasses\",\"geek\",\"hipster\"]}},\"required\":[\"tags\"],\"additionalProperties\":false,\"$schema\":\"http://json-schema.org/draft-07/schema#\"}.\nError message: [\n  {\n    \"code\": \"invalid_type\",\n    \"expected\": \"array\",\n    \"received\": \"undefined\",\n    \"path\": [\n      \"tags\"\n    ],\n    \"message\": \"Required\"\n  }\n]",
    "cause": {
      "issues": [
        {
          "code": "invalid_type",
          "expected": "array",
          "received": "undefined",
          "path": [
            "tags"
          ],
          "message": "Required"
        }
      ],
      "name": "ZodError"
    },
    "stack": "...",
    "value": {
      "type": "object",
      "properties": {
        "tags": {
          "type": "array",
          "items": [
            "glasses",
            "eyeglasses",
            "nerd glasses",
            "geek",
            "hipster"
          ]
        }
      },
      "required": [
        "tags"
      ],
      "additionalProperties": false,
      "$schema": "http://json-schema.org/draft-07/schema#"
    }
  }
}

[sgomez]

Hi @davidbarker

The zod error is because the model returns a json schema and not just a object, so detailed_description is inside properties and not in the root of the object. I guess your prompt/schema is forcing the model to return a JSOM schema. Can u provide a demo repository to check the issue?

Anyway I guess, is a problem with the model and not with the provider. This example works to me:

https://github.com/sgomez/ollama-ai-provider/blob/main/examples/ai-core/src/generate-object/ollama-multimodal.ts

[davidbarker]

Ahhh, that makes sense! Thanks. (And thanks for the prompt reply.)

I'll keep playing with it — I guess I could always extract the response data from the error message, and then process it myself.