JsonSchemaExporter.GetJsonSchemaAsNode "format" output cannot be controlled by configuration

Description

For numeric types, a regular expression is output using the "pattern" key, but this can be suppressed by setting JsonSerializerOptions.NumberHandling to JsonNumberHandling.Strict.
For date and time types, the value format is output using the "format" key.However, the format output is hard-coded and cannot be suppressed by configuration. I have confirmed similar implementations with DateOnly, DateTime, DateTimeOffset, TimeOnly, and Guid.

https://github.com/dotnet/runtime/blob/v9.0.0-preview.7.24405.7/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/Converters/Value/DateOnlyConverter.cs#L82

Similar to controlling the output of the "pattern" key through JsonSerializerOptions.NumberHandling, I believe JsonSchemaExporterOptions (which I think is more appropriate than JsonSerializerOptions) should have a configuration that allows you to change the "format" key output as needed.

var options = new JsonSchemaExporterOptions
{
    ValueFormatHandling = JsonValueFormatHandling.None, // or Always
};

Reproduction Steps

Uses System.Text.Json 9.0.0-preview.7.24405.7. Run the following code:

var serializerOptions = new JsonSerializerOptions(JsonSerializerDefaults.Web)
{
    NumberHandling = JsonNumberHandling.Strict,
    PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseLower,
    RespectNullableAnnotations = true,
    TypeInfoResolver = new DefaultJsonTypeInfoResolver(),
    WriteIndented = true
};

var exporterOptions = new JsonSchemaExporterOptions
{
    TreatNullObliviousAsNonNullable = true,
};

var schema = serializerOptions.GetJsonSchemaAsNode(typeof(TestModel), exporterOptions);
Console.WriteLine(schema.ToJsonString(serializerOptions));

public class TestModel
{
    public required DateOnly DateOnlyRequiredNotNull { get; set; }
    public required TimeOnly? TimeOnlyRequiredAllowNull { get; set; }
    public required DateTime DateTimeRequiredNotNull { get; set; }
    public required DateTimeOffset? DateTimeOffsetRequiredAllowNull { get; set; }
    public required Guid GuidRequiredNotNull { get; set; }
}

Expected behavior

{
  "type": "object",
  "properties": {
    "date_only_required_not_null": {
      "type": "string"
    },
    "time_only_required_allow_null": {
      "type": [
        "string",
        "null"
      ]
    },
    "date_time_required_not_null": {
      "type": "string"
    },
    "date_time_offset_required_allow_null": {
      "type": [
        "string",
        "null"
      ]
    },
    "guid_required_not_null": {
      "type": "string"
    }
  },
  "required": [
    "date_only_required_not_null",
    "time_only_required_allow_null",
    "date_time_required_not_null",
    "date_time_offset_required_allow_null",
    "guid_required_not_null"
  ]
}

Actual behavior

"format" is generated for each property, but it cannot be suppressed in the configuration.

{
  "type": "object",
  "properties": {
    "date_only_required_not_null": {
      "type": "string",
      "format": "date"
    },
    "time_only_required_allow_null": {
      "type": [
        "string",
        "null"
      ],
      "format": "time"
    },
    "date_time_required_not_null": {
      "type": "string",
      "format": "date-time"
    },
    "date_time_offset_required_allow_null": {
      "type": [
        "string",
        "null"
      ],
      "format": "date-time"
    },
    "guid_required_not_null": {
      "type": "string",
      "format": "uuid"
    }
  },
  "required": [
    "date_only_required_not_null",
    "time_only_required_allow_null",
    "date_time_required_not_null",
    "date_time_offset_required_allow_null",
    "guid_required_not_null"
  ]
}

Regression?

No response

Known Workarounds

Of course, I know I can change the generated JSON Schema by setting methods on JsonSchemaExporterOptions.TransformSchemaNode. In fact, I generated the Expected schema with the code below.

Console.WriteLine(OpenAIStructuredOutputs.CreateJsonSchema<TestModel>());

public static class OpenAIStructuredOutputs
{
    private static readonly JsonSerializerOptions s_serializerOptions = new(JsonSerializerDefaults.Web)
    {
#if DEBUG
        WriteIndented = true,
#endif
        NumberHandling = JsonNumberHandling.Strict,
        PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseLower,
        RespectNullableAnnotations = true,
        TypeInfoResolver = new DefaultJsonTypeInfoResolver()
    };

    private static readonly JsonSchemaExporterOptions s_exporterOptions = new()
    {
        TreatNullObliviousAsNonNullable = true,
        TransformSchemaNode = OnTransformSchemaNode
    };

    public static string CreateJsonSchema<T>()
    {
        var schema = s_serializerOptions.GetJsonSchemaAsNode(typeof(T), s_exporterOptions);
        return schema.ToJsonString(s_serializerOptions);
    }

    private static JsonNode OnTransformSchemaNode(JsonSchemaExporterContext context, JsonNode node)
    {
        if(context.TypeInfo.Kind != JsonTypeInfoKind.Object)
        {
            return node;
        }

        var current = node.AsObject();

        // It would be useful to have an option to always output "additionalProperties".
        if(!current.TryGetPropertyValue("additionalProperties", out _))
        {
            current.Add("additionalProperties", false);
        }

        // Search inside "properties" and delete the "format" key if it exists.
        if(current.TryGetPropertyValue("properties", out var properties))
        {
            if(properties is JsonObject propertiesObject)
            {
                for(int i = 0; i < propertiesObject.Count; i++)
                {
                    if(propertiesObject[i] is not JsonObject property)
                    {
                        continue;
                    }

                    if(property.TryGetPropertyValue("format", out _))
                    {
                        property.Remove("format");
                    }
                }
            }
        }

        return current;
    }
}

public class TestModel
{
    public required DateOnly DateOnlyRequiredNotNull { get; set; }
    public required TimeOnly? TimeOnlyRequiredAllowNull { get; set; }
    public required DateTime DateTimeRequiredNotNull { get; set; }
    public required DateTimeOffset? DateTimeOffsetRequiredAllowNull { get; set; }
    public required Guid GuidRequiredNotNull { get; set; }
}

I would like to generate a JSON schema to be passed to the Structured Outputs of OpenAI GPT-4o (2024-08-06) using GetJsonSchemaAsNode.

There are some restrictions on the JSON schema that GPT-4o accepts.

"pattern" and "format" cannot be specified
"required" is required for all fields
"additionalProperties" must be set to false for all objects etc.

I think the same problem will arise in the semantic kernel. Therefore, property generation control should be as configurable as possible.

Configuration

No response

Other information

No response

Tagging subscribers to this area: @dotnet/area-system-text-json, @gregsdennis See info in area-owners.md if you want to be subscribed.

[Just giving some JSON Schema context here. I have no comment on the generator implementation.]

It's important to note (explicitly, because people generally get this wrong) that format doesn't validate unless the tool has been explicitly configured to do so. It's informational only.

The spec says that a date-time format should inform the user to expect an RFC3339 date/time. But .Net uses the ISO8601 date/time format. Within that context, and since there's no validation of the format, it could be understood by a .Net consumer that the format should be ISO8601.

If the schema is going to be shared outside of a .Net ecosystem, then perhaps the RFC3339 requirement should be considered more carefully.

It's also allowable to create your own formats so long as consumers can be made to understand it. So you could create an iso8601 format (or whatever you want to name it).

For numeric types, a regular expression is output using the "pattern" key

I assume this is only for cases where a number is string-encoded? (I think encoding numbers as strings is a bad practice and is perpetuated only because parsers generally don't support numbers well.) pattern is only applied to string instances and won't affect numbers. Furthermore, no keyword can be applied to the JSON-encoding itself. JSON Schema operates on the JSON data model not on the JSON text itself.

Thanks for your comment. I think I understand that the "format" keyword is an annotation by default and does not affect actual validation. For the above numeric types, this is probably because the default value of the NumberHandling property in JsonSerializerDefaults.Web is JsonNumberHandling.AllowReadingFromString. That's why I think it would be good to be able to control not to output these annotations when generating a schema.

As I mentioned at the end of my first post, my intention with using JSON schema is to tell the AIto "conform to the specified structure fotmat" as its response. The actual format of the data exchange is a matter outside of JSON schema. Validating the answer is yet another matter, and you must first check whether it is a grammatically valid JSON document.

For example, the following prompt requests the creation of data that conforms to the JSON schema.

// Original prompt (Japanese)
IEnumerable<ChatMessage> messages = [
    new SystemChatMessage("内容は日本語で、応答形式で指定した JSON スキーマに対してデータを作成してください。"),
    new SystemChatMessage("ユーザーは日本から問い合わせているため、日時は JST（日本標準時）UTC+09:00 を利用し、ISO 8601 形式で出力してください。"),
    new UserChatMessage("null 許可の場合でも値を設定してください。"),
];

// In English
IEnumerable<ChatMessage> messages = [
    new SystemChatMessage("The content should be in Japanese, and the data should be created according to the JSON schema specified in the response format."),
    new SystemChatMessage("Since the user is making an inquiry from Japan, please use JST (Japan Standard Time) UTC+09:00 for the date and time, and output in ISO 8601 format."),
    new UserChatMessage("Please set a value even if null is allowed."),
];

Using the JSON schema described in "Expected behavior", the following response was returned, although no property definitions were specified.

{
    "date_only_required_not_null": "2023-10-09",
    "time_only_required_allow_null": "14:30:00",
    "date_time_required_not_null": "2023-10-09T14:30:00+09:00",
    "date_time_offset_required_allow_null": "2023-10-09T14:30:00+09:00",
    "guid_required_not_null": "123e4567-e89b-12d3-a456-426614174000"
}

AI assigns a time zone to date_time_required_not_null, but my implementation is of DateTime type, so it cannot be simply deserialized.

To give another example, in Japan, depending on the industry and context, "26:00:00" may be used to mean "2:00 AM the next day," and it may not be possible to simply deserialize it into TimeOnly.

I think these are some of the points you mentioned, as well as things to be careful about when sharing a schema outside the .NET ecosystem.

In particular, the AI side is a complete black box, so I'm groping around to see if I can get a response that meets my intentions.

https://platform.openai.com/docs/guides/structured-outputs/structured-outputs

Configuring the schema of individual types is something that isn't supported for the moment (primarily because we don't yet have a publicly accessible JsonSchema exchange type that something like a custom converter could try to configure). Right now, the only available mechanism for applying customizations is the post-hoc TransformSchemaNode delegate that you've already discovered.

The reason why you're seeing the pattern keyword in cases where JsonNumberHandling.AllowReadingFromString is because generator is producing schemas that are specifying the behavior of JsonSerializer. In other words, an instance should be valid under the schema if and only if it is accepted as input by the JsonSerializer.Deserialize methods for the same type. It is understandable that your application might require a schema document that is a stricter subset of what the deserializer accepts, and it should be possible to make such refinements with the transformer delegate as an interim solution.

There are some restrictions on the JSON schema that GPT-4o accepts.

"pattern" and "format" cannot be specified

"required" is required for all fields

"additionalProperties" must be set to false for all objects

etc.

To an extent it should be possible to influence your output by appropriately modelling your types directly:

The pattern keyword can be avoided in all configurations by setting a JsonNumberHandlingAttribute on the type level.
You can mark all your properties as required to influence the relevant keyword. Alternatively you could try using a deserialization constructor and then enable the JsonSerializerOptions.RespectRequiredConstructorParameters flag that has been added in .NET 9.
additionalProperties: false is added to all types if you have the JsonSerializerOptions.UnmappedMemberHandling property set to JsonUnmappedMemberHandling.Disallow.

I also commented on #107508, the generated JSON schema is intended to be used outside the .NET ecosystem (e.g., OpenAI GPT-4o).

Outside the ecosystem, the "format" keyword is an annotation by default and has no effect on actual validation. Therefore, the definition of "format" is an inside .NET implementation issue. Whether the external service validates the JSON data according to the "format" is also outside the scope of the schema definition.

So, I don't plan to directly pass the received data to JsonSerializer.Deserialize, but will do so while validating it using JsonDocument.

The current behavior is fine as the default within the .NET ecosystem, but it should have an option in the configuration to not output "format" for outside the .NET ecosystem.

additionalProperties: false is added to all types if you have the JsonSerializerOptions.UnmappedMemberHandling property set to JsonUnmappedMemberHandling.Disallow.

Good news! But the additionalProperties isn't output, am I doing something wrong?

var serializerOptions = new JsonSerializerOptions(JsonSerializerDefaults.Web)
{
  TypeInfoResolver = new DefaultJsonTypeInfoResolver(),
  UnmappedMemberHandling = JsonUnmappedMemberHandling.Disallow
};

var exporterOptions = new JsonSchemaExporterOptions { TreatNullObliviousAsNonNullable = true, };

var schema = serializerOptions.GetJsonSchemaAsNode(typeof(TestModel), exporterOptions);
Console.WriteLine(schema.ToJsonString(serializerOptions));

public class TestModel
{
  public required string Data { get; set; }
}

{"type":"object","properties":{"data":{"type":"string"}},"required":["data"]}

Outside the ecosystem, the "format" keyword is an annotation by default and has no effect on actual validation. Therefore, the definition of "format" is an inside .NET implementation issue.

I don't follow this reasoning. Why does format not being validated by default imply that format is a .NET implementation detail? That's not true at all.

The current behavior is fine as the default within the .NET ecosystem, but it should have an option in the configuration to not output "format" for outside the .NET ecosystem.

I still don't understand. The schema being generated is a valid JSON schema document that specifies the particular JsonSerializerOptions it has been passed. There is nothing .NET specific about the format keyword. The only issue I'm seeing is that Open AI impose restrictions in what schema documents their APIs accept. This is expected, and different vendors have different metaschematization requirements which is why we added TransformSchemaNode.

Good news! But the additionalProperties isn't output, am I doing something wrong?

I think you might have hit an actual bug where the global setting is being ignored. Try adding it as an attribute annotation instead:

[JsonUnmappedMemberHandling(JsonUnmappedMemberHandling.Disallow)]

I was successful when I added the [JsonUnmappedMemberHandling(JsonUnmappedMemberHandling.Disallow)] attribute. Thanks!

I wanted to use this library to generate a JSON schema to be used for the response_format of GPT-4o (2024-08-06) Structured Outputs in Azure OpenAI.

However, as I mentioned in the issues I created, the current default behavior returns 400 (Bad Request) as shown below, so I looked for configuration options for schema generation.

It seems better to implement TransformSchemaNode and adjust it without expecting standard support.

If JsonSchemaExporterOptions.TreatNullObliviousAsNonNullable is false.

Invalid schema for response_format 'TestModel': schema must be a JSON Schema of 'type: "object"', got 'type: "['object', 'null']"'.'

If a property don't have required keyword.

[JsonUnmappedMemberHandling(JsonUnmappedMemberHandling.Disallow)]
public class TestModel
{
    public string? StringAllowNull { get; set; }
}

Invalid schema for response_format 'TestModel': In context=(), 'required' is required to be supplied and to be an array including every key in properties. Missing 'string_allow_null''

If JsonSerializerOptions.UnmappedMemberHandling is not JsonUnmappedMemberHandling.Disallow.

// NG
public class TestModel
{
  public required DateOnly DateOnlyNotNull { get; set; }
}

// OK
[JsonUnmappedMemberHandling(JsonUnmappedMemberHandling.Disallow)]
public class TestModel
{
    public required DateOnly DateOnlyNotNull { get; set; }
}

Invalid schema for response_format 'TestModel': In context=(), 'additionalProperties' is required to be supplied and to be false'

If the property type is one of DateOnly, DateTime, DateTimeOffset, TimeOnly, and Guid.

[JsonUnmappedMemberHandling(JsonUnmappedMemberHandling.Disallow)]
public class TestModel
{
  public required DateOnly DateOnlyNotNull { get; set; }
}

Invalid schema for response_format 'TestModel': In context=('properties', 'date_only_not_null'), 'format' is not permitted'

If JsonSerializerOptions.NumberHandling is not JsonNumberHandling.Strict.

[JsonUnmappedMemberHandling(JsonUnmappedMemberHandling.Disallow)]
public class TestModel
{
  public required decimal DecimalNotNull { get; set; }
}

Invalid schema for response_format 'TestModel': In context=('properties', 'decimal_not_null'), 'pattern' is not permitted'

If the proprty type is enum with FlagsAttribute

[JsonUnmappedMemberHandling(JsonUnmappedMemberHandling.Disallow)]
public class TestModel
{
  public required FileAccess EnumWithFlagsNotNull { get; set; }
}

Generated Json schema:

{
  "type": "object",
  "properties": {
    "enum_with_flags_not_null": {
      "type": "string"
    }
  },
  "required": [
    "enum_with_flags_not_null"
  ],
  "additionalProperties": false
}

LLM outputs: ("default_value" is not defined in System.IO.FileAccess)

{"enum_with_flags_not_null":"default_value"}

If you can use GPT-4o with Azure OpenAI, you can use the following code:

// NuGet\Install-Package Azure.AI.OpenAI -Version 2.0.0-beta.5
// NuGet\Install-Package System.Text.Json -Version 9.0.0-preview.7.24405.7

// Uses the GPT-4o 2024-08-06 deployment
var config = new OpenAIConfig
{
  Endpoint = "",
  ApiKey = "",
  Deployment = ""
};

// Change API version for testing.
// OpenAI 2.0.0-beta9 supports thr Structured Outputs,
// but Azure.AI.OpenAI 2.0.0-beta.5 doesn't support 2024-08-01-preview
var option = new AzureOpenAIClientOptions();
var fi = typeof(AzureOpenAIClientOptions)
  .GetTypeInfo()
  .DeclaredFields
  .First(r => r.FieldType == typeof(string) && r.Name != "_applicationId");
fi.SetValue(option, "2024-08-01-preview");

var azureOpenAiClient = new AzureOpenAIClient(new Uri(config.Endpoint), new ApiKeyCredential(config.ApiKey), option);
var chatClient = azureOpenAiClient.GetChatClient(config.Deployment);

var schema = GenerateJsonSchema<TestModel>();
Console.WriteLine(schema);

var requestOptions = new ChatCompletionOptions
{
  ResponseFormat = ChatResponseFormat.CreateJsonSchemaFormat(nameof(TestModel), BinaryData.FromString(schema), null, true)
};

IEnumerable<ChatMessage> messages = [
  new SystemChatMessage("The content should be in Japanese, and the data should be created according to the JSON schema specified in the response format."),
  new SystemChatMessage("Since the user is making an inquiry from Japan, please use JST (Japan Standard Time) UTC+09:00 for the date and time, and output in ISO 8601 format."),
  new UserChatMessage("Please set a value even if null is allowed."),
];

var response = await chatClient.CompleteChatAsync(messages, requestOptions);

// I have omitted null checks and json string validation.
var data = response.Value.Content[0].Text;
Console.WriteLine(data);

string GenerateJsonSchema<T>()
{
  var serializerOptions = new JsonSerializerOptions(JsonSerializerDefaults.Web)
  {
    WriteIndented = true,
    NumberHandling = JsonNumberHandling.Strict,
    Converters = { new JsonStringEnumConverter() },
    PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseLower,
    RespectNullableAnnotations = true,
    TypeInfoResolver = new DefaultJsonTypeInfoResolver(),
    UnmappedMemberHandling = JsonUnmappedMemberHandling.Disallow
  };

  var exporterOptions = new JsonSchemaExporterOptions
  {
    TreatNullObliviousAsNonNullable = true
  };

  var schemaNode = serializerOptions.GetJsonSchemaAsNode(typeof(T), exporterOptions);
  return schemaNode.ToJsonString(serializerOptions);
}

[JsonUnmappedMemberHandling(JsonUnmappedMemberHandling.Disallow)]
public class TestModel
{
  public required string? StringAllowNull { get; set; }
}

public class OpenAIConfig
{
  public required string Endpoint { get; set; }
  public required string ApiKey { get; set; }
  public required string Deployment { get; set; }
}

dotnet / runtime