Error undefined (reading 'url') with GPT-Preview with Images

[JpEncausse]

• Package Name: azure/openai
• Package Version: ^1.0.0-beta.12"
• Operating system: Ubuntu
• [x] nodejs
  • version: 20

Describe the bug Following the documentation on ChatGPT-Vision Preview I got the message from OpenAI TypeError: Cannot read properties of undefined (reading 'url')

To Reproduce Steps to reproduce the behavior:

1. Setup the model "vision-preview" on Sweden Tenant
2. Run on Azure OpenAI Studio : Upload an Image + Text => It Works
3. Run the same code with Client => got the error

Expected behavior Should return a description of the image and/or a better error description

Additional context I run this code on Node-RED. Everything works with various Model and Text but not with image.

let chat = [
    {
        "role": "system",
        "content": [
            {
                "type": "text",
                "text": "You are an AI assistant that helps people find information."
            }
        ]
    },
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "What is this"
            },
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
                }
            }]
    }]

try {
    msg.payload = await client.getChatCompletions(deploymentId, chat);
    return [undefined, msg];

} catch(err){
    msg.payload = { err, chat, }
    return [msg, undefined];
}

StackTrace

TypeError: Cannot read properties of undefined (reading 'url')
    at serializeChatMessageImageContentItem (/node_modules/@azure/openai/dist/index.cjs:244:39)
    at serializeChatRequestContentItemUnion (/node_modules/@azure/openai/dist/index.cjs:206:20)
    at Array.map (<anonymous>)
    at serializeChatRequestUserMessage (/node_modules/@azure/openai/dist/index.cjs:198:30)
    at serializeChatRequestMessageUnion (/node_modules/@azure/openai/dist/index.cjs:231:20)
    at /node_modules/@azure/openai/dist/index.cjs:701:51
    at Array.map (<anonymous>)
    at _getChatCompletionsSend (/node_modules/@azure/openai/dist/index.cjs:701:40)
    ...

[github-actions[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @glharper.

[JpEncausse]

According to the code there is an error with variable : image_url versus imageUrl

return {
        type: obj["type"],
        image_url: { url: obj.imageUrl["url"], detail: obj.imageUrl["detail"] },
};

Replacing the query with camelcase fix the issue

{
  "type": "image_url",
  "imageUrl": {
    "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
  }
}

[JpEncausse]

Setting options in getChatCompletions() will send the image and text but trigger also errors. Removing all options Fix it but ... how do you know if the user set an image ??

I use :

const options = {
    "temperature": 0.7,
    "top_p": 0.95,
    "frequency_penalty": 0,
    "presence_penalty": 0,
    "max_tokens": 800,
    "stop": null,
    "tools": msg.llm.tools,
};

It is the stop attribute that trigger the error from server

[deyaaeldeen]

I am glad this issue has been resolved. We strongly recommend that you migrate to using the AzureOpenAI client exported by the official OpenAI client library for JavaScript. Please checkout the migration guide for information on how to do so.

[JpEncausse]

@deyaaeldeen I was using the latest version (from NPM @azure/openai) and the migration doc you point sems to be a beta version ?

[deyaaeldeen]

@JpEncausse To give more context, the latest of @azure/openai is v2.0.0-beta.1: https://www.npmjs.com/package/@azure/openai/v/2.0.0-beta.1. The migration guide asks you to migrate from @azure/openai@1.0.0-beta.12 to both [openai@^4.42.0] (https://www.npmjs.com/package/openai) and @azure/openai@1.0.0-beta.1. Please let me know if you have any questions about the migration process.