[BUG] Azure OpenAI Client fails with 500 Internal Server Error for Image Message Content

[meichenberger78]

Library name and version

Azure.AI.OpenAI 2.0.0-beta.2

Describe the bug

When using the Azure OpenAI Client, text-only requests work as expected. However, when attempting to include an image message content part in the request, the client fails with a 500 Internal Server Error after approximately 30-40 seconds.

Expected behavior

The request should successfully return a response describing the image.

Actual behavior

The request fails with a 500 Internal Server Error after 30-40 seconds.

Reproduction Steps

var openAiClient = new AzureOpenAIClient(
    new Uri(ENDPOINT),
    new AzureKeyCredential(APIKEY)
);
var client = openAiClient.GetChatClient(MODELNAME); //gtp-4o Model

// Text-only request - works fine
var response = await client.CompleteChatAsync(
    [
        new UserChatMessage(
            ChatMessageContentPart.CreateTextMessageContentPart("Hello, how are you?")
        )
    ]
);
Console.WriteLine(response.Value.Content[0].Text);

using var httpClient = new HttpClient();
var imageBytes = await httpClient.GetByteArrayAsync("https://en.wikipedia.org/wiki/Microsoft#/media/File:Aerial_Microsoft_West_Campus_August_2009.jpg");

// Service request failed. Status: 500(Internal Server Error)
response = await client.CompleteChatAsync(
    [
        new UserChatMessage(
            ChatMessageContentPart.CreateTextMessageContentPart("Describe the image"),
            ChatMessageContentPart.CreateImageMessageContentPart(new BinaryData(imageBytes), "image/png", ImageChatMessageContentPartDetail.Low)
        )
    ]
);

Environment

Windows. .Net 8 Modellname: gpt-4o Modellversion: 2024-05-13 Location: WestEurope

[github-actions[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @jpalvarezl @ralph-msft @trrwilson.

[trrwilson]

Thanks, @meichenberger78 -- just to isolate, can you attempt with a local file, e.g. via File.ReadAllBytes()?

The link in the example code isn't a direct image link (rather, an HTML view framing it) and so I'm seeing a different error when attempting a direct repro:

Unhandled exception. System.ClientModel.ClientResultException: HTTP 400 (BadRequest)

Invalid image data.

The long wait and final HTTP 500 isn't good and is different from the prompt HTTP 400 I see, but using direct image bytes may get things running as intended.

[meichenberger78]

@trrwilson thank you and sorry my fault!