[BUG] Faulty serialization/deserialization of messages in ChatCompletionsOptions

Library name and version

Azure.AI.OpenAI 1.0.0-beta.13

Describe the bug

Me and my team are building a custom application on top of the Azure OpenAI Services service with the gpt-4-32k model deployment.

We initially used the Azure.AI.OpenAI SDK with version 1.0.0-beta.5 to facilitate the communication with Azure OpenAI Services. Then our stakeholder made a feature request to enable ChatGPT to answer questions based on our own data. We discovered the BYOD feature where we could upload our data to a Storage Account, index it using Azure AI Search and Semantic ranker, and then query the data through ChatGPT in Azure OpenAI Services. Problem is, we couldn't get it to work with SDK Azure.AI.OpenAI, version=1.0.0-beta.5.

So we decided to reverse engineer the whole thing, aka we created our own classes matching the requests and responses that we could see were being made over the network while we were using the ChatGPT Playground. Then we used these classes in HTTP requests to our Azure OpenAI Services service with the correct headers and payloads. It's not beautiful, but its working.

Now we need to implement the streaming API in the Azure.AI.OpenAI SDK, but when we tried the normal req/resp API in the SDK we found some serious problems in the serialization logic.

The following is a console application that takes the example from the SDKs README file on nuget.org, serializes it and then writes the output to the console:

using System.Text.Json;
using System.Text.Json.Serialization;
using Azure.AI.OpenAI;

var chatCompletionsOptions = new ChatCompletionsOptions()
{
    DeploymentName = "gpt-3.5-turbo", // Use DeploymentName for "model" with non-Azure clients
    Messages =
    {
        // The system message represents instructions or other guidance about how the assistant should behave
        new ChatRequestSystemMessage("You are a helpful assistant. You will talk like a pirate."),
        // User messages represent current or historical input from the end user
        new ChatRequestUserMessage("Can you help me?"),
        // Assistant messages represent historical responses from the assistant
        new ChatRequestAssistantMessage("Arrrr! Of course, me hearty! What can I do for ye?"),
        new ChatRequestUserMessage("What's the best way to train a parrot?"),
    }
};

var opt = JsonSerializer.Serialize(chatCompletionsOptions);
Console.WriteLine(opt);

It gives the following output in the console:

{
    "ChoiceCount": null,
    "DeploymentName": "gpt-3.5-turbo",
    "FrequencyPenalty": null,
    "MaxTokens": null,
    "NucleusSamplingFactor": null,
    "PresencePenalty": null,
    "StopSequences": [],
    "Temperature": null,
    "TokenSelectionBiases": {},
    "Functions": [],
    "FunctionCall": null,
    "AzureExtensionsOptions": null,
    "ToolChoice": null,
    "Messages": [
        {
            "Role": {}
        },
        {
            "Role": {}
        },
        {
            "Role": {}
        },
        {
            "Role": {}
        }
    ],
    "User": null,
    "Seed": null,
    "ResponseFormat": null,
    "Tools": []
}

See how the Messages property is missing both the roles and the messages?

I also tried creating a very simple API that takes the ChatCompletionsOptions object as a parameter and just writes it back to callers response message:

using Microsoft.AspNetCore.Mvc;
using Azure.AI.OpenAI;

var builder = WebApplication.CreateBuilder(args);
builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();

var app = builder.Build();

if (app.Environment.IsDevelopment())
{
    app.UseSwagger();
    app.UseSwaggerUI();
}

app.UseHttpsRedirection();

app.MapPost("/gpt", ([FromBody] ChatCompletionsOptions opt) =>
{
    return Results.Ok(opt);
})
.WithName("GPT")
.WithOpenApi();

app.Run();

I send this request:

{
    "messages": [
        {
            "role": "system",
            "content": "You are an AI assistant that helps people find information."
        },
        {
            "role": "user",
            "content": "Hi! How are you?"
        }
    ],
    "temperature": 0.7,
    "top_p": 0.95,
    "frequency_penalty": 0,
    "presence_penalty": 0,
    "max_tokens": 800,
    "stream": false
}

And I get this response back:

{
    "choiceCount": null,
    "deploymentName": null,
    "frequencyPenalty": null,
    "maxTokens": null,
    "nucleusSamplingFactor": null,
    "presencePenalty": null,
    "stopSequences": [],
    "temperature": 0.7,
    "tokenSelectionBiases": {},
    "functions": [],
    "functionCall": null,
    "azureExtensionsOptions": null,
    "toolChoice": null,
    "messages": [],
    "user": null,
    "seed": null,
    "responseFormat": null,
    "tools": []
}

The property messages is now empty. The same request works just fine when sent directly to the Azure OpenAI Services API using Postman. I guess this means that the ChatGPT Playground in Azure OpenAI Services doesn't use the Azure.AI.OpenAI .NET SDK.

With all that said, we need to move forward with our implementation of the streaming API, and we're not very happy about the idea to reverse engineer that one. Is there any chance that you can fix this issue anytime soon? Or might there be some specific version of the SDK that you know is working as expected (including CognotiveSearch extension), that we can use while waiting?

Expected behavior

The roles and messages in the conversation should be serialized properly so that they are included in the ChatCompletionsOptions request

Actual behavior

Roles and messages are being omitted in the serialization made by the SDK.

Reproduction Steps

Reproduction in Console App

Run the following code with .NET SDK installed on your machine:

using System.Text.Json;
using System.Text.Json.Serialization;
using Azure.AI.OpenAI;

var chatCompletionsOptions = new ChatCompletionsOptions()
{
    DeploymentName = "gpt-3.5-turbo", // Use DeploymentName for "model" with non-Azure clients
    Messages =
    {
        // The system message represents instructions or other guidance about how the assistant should behave
        new ChatRequestSystemMessage("You are a helpful assistant. You will talk like a pirate."),
        // User messages represent current or historical input from the end user
        new ChatRequestUserMessage("Can you help me?"),
        // Assistant messages represent historical responses from the assistant
        new ChatRequestAssistantMessage("Arrrr! Of course, me hearty! What can I do for ye?"),
        new ChatRequestUserMessage("What's the best way to train a parrot?"),
    }
};

var opt = JsonSerializer.Serialize(chatCompletionsOptions);
Console.WriteLine(opt);

Reproduction in Web API

Run the following code with .NET SDK installed on your machine:

using Microsoft.AspNetCore.Mvc;
using Azure.AI.OpenAI;

var builder = WebApplication.CreateBuilder(args);
builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();

var app = builder.Build();

if (app.Environment.IsDevelopment())
{
    app.UseSwagger();
    app.UseSwaggerUI();
}

app.UseHttpsRedirection();

app.MapPost("/gpt", ([FromBody] ChatCompletionsOptions opt) =>
{
    return Results.Ok(opt);
})
.WithName("GPT")
.WithOpenApi();

app.Run();

Use an HTTP client like Postman to send the following request:

{
    "messages": [
        {
            "role": "system",
            "content": "You are an AI assistant that helps people find information."
        },
        {
            "role": "user",
            "content": "Hi! How are you?"
        }
    ],
    "temperature": 0.7,
    "top_p": 0.95,
    "frequency_penalty": 0,
    "presence_penalty": 0,
    "max_tokens": 800,
    "stream": false
}

Environment

.NET SDK version: 8.0.101 OS: Ubuntu 22.04 IDE and version: Visual Studio Code 1.86.1

Azure / azure-sdk-for-net