ChatResponseFormat.CreateJsonSchemaFormat() is not working correctly.

[helloSalmon]

Confirm this is not an issue with the underlying OpenAI API

• [X] This is an issue with the Python library

Confirm this is not an issue with Azure OpenAI

• [X] This is not an issue with Azure OpenAI

Describe the bug

When I try to set response_format by ChatResponseFormat.CreateJsonSchemaFormat, but the variable's base is not created correctly. The fact that 'Schema' and 'Name' is input correctly can be checked with below capture.

To Reproduce

Use the code below with c# 9.0 and .NET 4.7.1 and use Debug mod to check it.

Code snippets

[System.Serializable]
public class ExploererFormat
{
    public string name { get; set; }
    public int age { get; set; }

    [Description("The character's background story.")]
    public string background { get; set; }
}

public class RandomExploerer : MonoBehaviour
{
    private ChatClient chatClient = new(model: "gpt-4o-2024-08-06", Environment.GetEnvironmentVariable("OPENAI_API_KEY"));

    private static string schemaString = null;

    // Start is called before the first frame update
    void Start()
    {
        CreateSchema();

        CreateRandom();
    }

    public void CreateSchema()
    {
        JSchemaGenerator generator = new JSchemaGenerator();
        generator.DefaultRequired = Required.Always; // required by OpenAI

        JSchema schema = generator.Generate(typeof(ExploererFormat));
        schema.AllowAdditionalProperties = false; // required by OpenAI

        schemaString = schema.ToString();

        Debug.Log(schemaString);
    }

    public async void CreateRandom()
    {
        BinaryData data = BinaryData.FromString(schemaString);
        if (data == null ) { Debug.Log("Why this is empty?"); }

        ChatResponseFormat format = ChatResponseFormat.CreateJsonSchemaFormat(                
            name: "Character_Sheet",
            jsonSchema: BinaryData.FromString(schemaString),
            strictSchemaEnabled: true);

        var superMessages = new List<ChatMessage>
        {
            ChatMessage.CreateUserMessage("As a novel writer, create a random novel character with the following instructions: 1. All outputs must be written in Korean. 2. Genre of the novel is Ocean Adventure.")
        };

        ChatCompletionOptions jsonOptions = ModelReaderWriter.Read<ChatCompletionOptions>(BinaryData.FromString("{ \"messages\": [{\"role\": \"user\", \"content\": \"hello world\"}] }"));
        jsonOptions.ResponseFormat = format;
        Debug.Log(ModelReaderWriter.Write(jsonOptions).ToString());

        ChatCompletion chatCompletion = await chatClient.CompleteChatAsync(        
            superMessages
            ,jsonOptions);
}

OS

winOS

.NET version

4.7.1

Library version

2.0.0-beta.9

[joseharriaga]

Thank you for reaching out, @helloSalmon ! To diagnose the issue, I'm trying to understand if there might be a problem with the JsonSchemaGenerator that you're using. I just tried the following in a .NET Framework 4.7.2 solution, and it works for me:

ChatClient client = new ChatClient(
    "gpt-4o-2024-08-06",
    new ApiKeyCredential(Environment.GetEnvironmentVariable("OPENAI_API_KEY")));

string schema = 
    "{ "
    + "\"type\": \"object\","
    +     "\"properties\": {"
    +         "\"name\": { \"type\": \"string\" },"
    +         "\"age\": { \"type\": \"integer\" },"
    +         "\"background\": {"
    +             "\"type\": \"string\","
    +             "\"description\": \"The character's background story.\""
    +         "}"
    +     "},"
    +     "\"required\": [\"name\", \"age\", \"background\"],"
    +     "\"additionalProperties\": false"
    + "}";

ChatResponseFormat format = ChatResponseFormat.CreateJsonSchemaFormat(
    name: "Character_Sheet",
    jsonSchema: BinaryData.FromString(schema),
    strictSchemaEnabled: true);

ChatCompletionOptions options = new ChatCompletionOptions()
{
    ResponseFormat = format
};

List<ChatMessage> messages = new List<ChatMessage>
{
    new UserChatMessage("As a novel writer, create a random character for an ocean adventure novel.")
};

ChatCompletion chatCompletion = client.CompleteChat(messages, options);

Console.WriteLine(chatCompletion.Content[0].Text);

I got the following output:

{"name":"Meredith Wavesong","age":32,"background":"Meredith Wavesong grew up in the small coastal town of Starfish Bay, where the salty scent of the ocean was the melody that accompanied her childhood. As the only child of two marine biologists, she found herself more comfortable in the water than on land, spending most of her days exploring tide pools, collecting seashells, and learning about marine life from her parents.\n\nAt the age of 17, a fateful diving expedition with her parents took a tragic turn when a sudden storm engulfed their boat. Meredith was the sole survivor, washing ashore on a deserted stretch of land she did not recognize. For weeks, she survived by tapping into the survival skills her parents once taught her, piecing together remnants of florid tales about ancient oceanic legends that her parents had narrated by moonlight.\n\nYears later, the marine community buzzed with whispers about a rare aquatic whisperer who could communicate with marine animals. It was believed that Meredith had developed an uncanny intuition for underwater communication during her days alone on the mysterious island, forever forming a bond with the watery realms. Her once-silver hair was now streaked with the faint colors of coral and seabreeze, changes she herself could not explain.\n\nNow, at 32, Meredith has become something of a nomad, following the ocean's call wherever it takes her, her heart yearning for the answers to the secrets and treasures that lie beneath the waves. She volunteers on ocean conservation missions, sometimes on the brink of being kicked off due to her spontaneous antics and unpredictable nature, driven by an unquenchable passion for adventure and discovery.\n\nArmed with her curious ability, an array of carefully curated diving gear, and an unyielding spirit, Meredith finds herself on the cusp of an unprecedented oceanic expedition. She hears of an ancient ship said to be haunted by the spirits of the drowned and whose contents could unlock secrets from the depths of history and the future of humanity.\n\nFor Meredith, the ocean isn't just a place; it's a calling, a guardian. She feels an inexplicable pull toward its deepest mysteries, and like the tides, she knows she must go where the currents lead her."}

Could you confirm if this works for you too?

[helloSalmon]

Thank you for your reply. Actually, this project is being built with Unity.

When I test it with Unity, this doesn't work. However, I created a entirely new project that works completely independently of Unity and tested it, and it worked fine as you suggested. Unity library(or maybe Rosyln compiler that Unity built in-house) seems to have been the problem.

Thanks to your help, I was able to figure out the cause of the problem. Thank you so much.

[joseharriaga]

Excellent! I'm glad to hear that. 😊

[Swah]

@helloSalmon I'm facing an that could be similar to yours (using Unity here as well). I didn't see your original message though so it's hard to tell.

The issue I'm having is

Couldn't get response from ChatGPT: System.ClientModel.ClientResultException: HTTP 400 (invalid_request_error: missing_required_parameter)
Parameter: response_format.json_schema
Missing required parameter: 'response_format.json_schema'.
  at OpenAI.ClientPipelineExtensions.ProcessMessageAsync (System.ClientModel.Primitives.ClientPipeline pipeline, System.ClientModel.Primitives.PipelineMessage message, System.ClientModel.Primitives.RequestOptions options)

Was yours similar, and if so how did you fix in Unity?

[helloSalmon]

@Swah Actually, I try strctured outputs via function calling instead of response_format:

public class ExploererFormat
{
    public string name { get; set; }
    public int age { get; set; }

    [Description("The character's background story.")]
    public string background { get; set; }
}

private static readonly ChatTool getSchema = ChatTool.CreateFunctionTool(
        functionName: nameof(ExploererFormat),
        functionDescription: "This is a format to create novel charater which user wants.",
        functionParameters: BinaryData.FromString(CreateSchema())
        );

ChatCompletionOptions options = new()
        {
            Tools = { getSchema },
            ToolChoice = ChatToolChoice.Required,
        };

This worked for me also in Unity. I hope this would helpful to you.

[Swah]

Thanks for your help @helloSalmon - it didn't work right away for me, but the response seems different so I'll debug further and report back here. Good to know that you didn't get it to work with the usual chat though, so there could still be a real issue here.

[Swah]

Alright interesting - using function calls also works for me in unity. I'm not sure what potential problems we're adding by strictly using function calls, so this seems like a problem worth solving.