useChat setMessages Assertion

[jongreen96]

Description

useChat's chat helper setMessages errors when setting a messages content to anything other than a string.

Error: Assertion: Unexpected value [object Object] for children prop, expected string

This is incorrect behavior as OpenAI's API accepts a string or array when working with their vision models, I have included a example below.

messages: [
      {
        role: "user",
        content: [
          { type: "text", text: "What’s in this image?" },
          {
            type: "image_url",
            image_url:
              "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg",
          },
        ],
      },
    ],

Code example

setMessages([
    ...messages,
        {
            role: 'user',
            content: [
                 { type: 'text', text: input },
                 {
                     type: 'image_url',
                     image_url: { url: images[0], detail: 'low' },
                 },
           ],
       },
]);

Additional context

No response

[bioshazard]

Looks like its just a limitation of the Type, yea. Able to send this image payload without issue if I ignore the type. Just need to extend the type to include MessageContentComplex