dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.27k stars 4.73k forks source link

Support customizing enum member names in System.Text.Json #74385

Closed jscarle closed 3 months ago

jscarle commented 2 years ago

API Proposal

Concerning the question of adding built-in enum name customization support, this would need to be done via a new attribute type:

namespace System.Text.Json.Serialization;

[AttributeUsage(AttributeTargets.Field, AllowMultiple = false)]
public class JsonStringEnumMemberNameAttribute : Attribute
{
    public JsonStringEnumMemberNameAttribute(string name);
    public string Name { get; }
}

API Usage

Setting the attribute on individual enum members can customize their name

JsonSerializer.Serialize(MyEnum.Value1 | MyEnum.Value2); // "A, B"

[Flags, JsonConverter(typeof(JsonStringEnumConverter))]
public enum MyEnum
{
    [JsonStringEnumMemberName("A")]
    Value1 = 1,

    [JsonStringEnumMemberName("B")]
    Value2 = 2,
}
Original Post [Issue 31081](https://github.com/dotnet/runtime/issues/31081) was closed in favor of [issue 29975](https://github.com/dotnet/runtime/issues/29975), however the scope of the issues differ. Issues 29975 is a discussion regarding the `DataContract` and `DataMember` attributes in general. Although `JsonStringEnumConverter `does address the straight conversion between an enum and its direct string representation, it does not in fact address cases where the string is not a direct match to the enum value. The following enum will NOT convert properly using the current implementation of `JsonStringEnumConverter`: ```csharp [JsonConverter(typeof(JsonStringEnumConverter))] public enum GroupType { [EnumMember(Value = "A")] Administrator, [EnumMember(Value = "U")] User } ``` ## Suggested Workaround See [this gist](https://gist.github.com/eiriktsarpalis/2c11d8dde598eab1b54281bc67a3df41) for a recommended workaround that works for both AOT and reflection-based scenaria.
ghost commented 2 years ago

Tagging subscribers to this area: @dotnet/area-system-text-json, @gregsdennis See info in area-owners.md if you want to be subscribed.

Issue Details
[Issue 31081](https://github.com/dotnet/runtime/issues/31081) was closed in favor of [issue 29975](https://github.com/dotnet/runtime/issues/29975), however the scope of the issues differ. Issues 29975 is a discussion regarding the `DataContract` and `DataMember` attributes in general. Although `JsonStringEnumConverter `does address the straight conversion between an enum and its direct string representation, it does not in fact address cases where the string is not a direct match to the enum value. The following enum will NOT convert properly using the current implementation of `JsonStringEnumConverter`: ``` [JsonConverter(typeof(JsonStringEnumConverter))] public enum GroupType { [EnumMember(Value = "A")] Administrator, [EnumMember(Value = "U")] User } ``` The solution proposed by JasonBodley works correctly: https://github.com/dotnet/runtime/issues/31081#issuecomment-848697673 ``` [JsonConverter(typeof(JsonStringEnumConverterEx))] public enum GroupType { [EnumMember(Value = "A")] Administrator, [EnumMember(Value = "U")] User } ``` However, this functionality should be built-in to the `JsonStringEnumConverter`.
Author: jscarle
Assignees: -
Labels: `area-System.Text.Json`, `untriaged`
Milestone: -
eiriktsarpalis commented 2 years ago

As mentioned in the issues you're linking to, it is unlikely we would add support for EnumMemberAttribute specifically, for the same reason that we are reluctant to do it for all the other attributes under System.Runtime.Serialization. These pertain to different/older serialization stacks and as such supporting them OOTB in System.Text.Json is not something we are planning to do.

We might consider exposing a dedicated attribute in the future, assuming there is substantial demand. Alternatively, it should be possible to add support using a custom converter, as has already been proposed by the community.

jscarle commented 2 years ago

Looking at the System.Text.Json.Serialization namespace, in addition to the JsonConverter and JsonConverter<T>, the only other JsonConverter added was the JsonStringEnumConverter. In other words, it was clearly identified that there is a frequent use case for converting between json string values and their enum equivalents, and vice-versa. However, right from the very beginning, people ran into situations where the string value is not necessarily the exact name of the enum, as it has been clearly demonstrated on stackoverflow: https://stackoverflow.com/questions/59059989/system-text-json-how-do-i-specify-a-custom-name-for-an-enum-value

I do agree that EnumMember is not the correct attribute, and that moving forward the goal should be to build upon System.Text.Json and not another unrelated namespace.

In the spirit of what's already been established with JsonStringEnumConverter, I would propose the following attribute:

namespace System.Text.Json.Serialization
{
    /// <summary>
    /// Specifies the enum string value that is present in the JSON when serializing and deserializing.
    /// </summary>
    [AttributeUsage(AttributeTargets.Field, Inherited = false, AllowMultiple = false)]
    public sealed class JsonStringEnumValueAttribute : JsonAttribute
    {
        /// <summary>
        /// Initializes a new instance of <see cref="JsonStringEnumValueAttribute"/> with the specified string value.
        /// </summary>
        /// <param name="value">The string value of the enum.</param>
        public JsonStringEnumValueAttribute(string value)
        {
            Value = value;
        }

        /// <summary>
        /// The string value of the enum.
        /// </summary>
        public string Value { get; }
    }
}

Which could then be used by the JsonStringEnumConverter specifically. Extending support for this attribute would eliminate the need for workarounds and nuget packages that people currently use and allow JsonStringEnumConverter to be used as the built-in go-to for all string to enum use cases.

eiriktsarpalis commented 2 years ago

Extending support for this attribute would eliminate the need for workarounds and nuget packages that people currently use

I should clarify that implementing all functionality OOTB is not a design goal for System.Text.Json. We do want to make sure the library offers the right extensibility points so that third party extensions can be defined easily and be successfully, however.

jscarle commented 2 years ago

I do agree, and there are popular alternatives such as Newtonsoft.Json which can fulfill the need for more advanced JSON serialization and deserialization. Its simply that the functionality for JsonStringEnumConverter has already been included in System.Text.Json, it would be nice to round off that functionality with this last missing bit.

String to enum conversions is a frequent use case, and due to the language restrictions for enum members, the occurrences where the string representation of the enum value will differ from the enum member name is quite frequent. Quite often due to the use of spaces, dashes, or underscores in the JSON string value used by the system with which .NET code may be communicating.

An alternative would be to enhance the parsing logic of Enum.TryParse (which is eventually called by JsonStringEnumConverter) to compensate for spaces, dashes, and underscores, but being that it's such a core method in the runtime, I doubt that would ever see the light of day.

IanKemp commented 2 years ago

Extending support for this attribute would eliminate the need for workarounds and nuget packages that people currently use

I should clarify that implementing all functionality OOTB is not a design goal for System.Text.Json. We do want to make sure the library offers the right extensibility points so that third party extensions can be defined easily and be successfully, however.

Do "the right extensibility points" include "hiding JsonStringEnumConverter's functionality in EnumConverter<T>, an internal sealed class that precludes said functionality from being easily extended by developers, thus forcing them to re-implement that same functionality entirely from scratch"?

Why is it that every time I need to do something simple like this feature request in STJ, I end up spending half a day looking for a solution, which turns out to be the opposite of simple... and along the way I find a similar solution for the same requirement in Newtonsoft.Json, and it is absolutely simple there? I very much get the impression that the STJ APIs and classes were designed by people who have never had to try to use STJ, whereas Newtonsoft.Json was written by a developer for developers.

To put it simply, Microsoft: if you want people to use STJ, why do you continually make it so difficult to use STJ? Why do you make me regret my decision to use it every time I try to use it? Why does this API have to be so unnecessarily, continually painful?

layomia commented 2 years ago

The contract resolver feature new in 7.0 could be extended to support user detection of EnumMemberAttribute with minimal configuration code to apply it to enum contracts. We can evaluate the feasibility of enabling this scenario in 8.0.

Maximys commented 1 year ago

@eiriktsarpalis , I want to provide some links about current Issue:

  1. System.Text.Json.Tests.EnumConverterTests.DuplicateNameEnumTest;
  2. System.Text.Json.Tests.EnumConverterTests.DuplicateNameEnum.FooBar.

Yes, may be I does not understand something, but this test and enum assume the implementation of EnumMemberAttribute support. Today I had alot of hadacke with JsonSerializer and EnumMemberAttribute. Can I try to implement this support?

bunnyi116 commented 1 year ago

Original content

我认为 JsonStringEnumConverter 应该内置一个自定义名称功能,因为 JsonConverter 使用枚举属性的字符串。

使用 JsonStringEnumConverter 时,应该检查 JsonPropertyNameAttribute 属性,如果有,则使用属性JsonPropertyName中的名称

对于 Enum 属性名称特性命名,可以直接使用 JsonPropertyNameAttribute 统一属性名称代码样式,而不应使用 EnumMemberAttribute

注:由于我的英语不好,我使用了机器翻译。如果有翻译错误,请原谅我。

Machine translation

I think JsonStringEnumConverter should have a custom name feature built in, because JsonConverter uses strings for enum properties.

When using JsonStringEnumConverter, you should check the JsonPropertyNameAttribute attribute and use the name from the attribute JsonPropertyName if present

For Enum property name attribute naming, you can directly use JsonPropertyNameAttribute to unify the property name code style instead of EnumMemberAttribute

Note: Since my English is not good, I used machine translation. Forgive me if there are translation errors.

bunnyi116 commented 1 year ago

The contract resolver feature new in 7.0 could be extended to support user detection of EnumMemberAttribute with minimal configuration code to apply it to enum contracts. We can evaluate the feasibility of enabling this scenario in 8.0.

Original content

我希望Attribute是这个 JsonPropertyNameAttribute,而不是EnumMemberAttribute。

因为我觉得JsonPropertyNameAttribute代表的是Json属性名称,所以有关Json序列化和反序列化的属性名称应该使用JsonPropertyNameAttribute。

注:由于我的英语不好,我使用了机器翻译。如果有翻译错误,请原谅我。

Machine translation

I want the Attribute to be this JsonPropertyNameAttribute instead of EnumMemberAttribute.

Because I think JsonPropertyNameAttribute represents the Json property name, so the Property name for Json serialization and deserialization should use JsonPropertyNameAttribute.

Note: Since my English is not good, I used machine translation. Forgive me if there are translation errors.

bunnyi116 commented 1 year ago

Interim programme

Temporarily available methods

Converter

public class JsonStringEnumConverter : JsonConverterFactory
{
    public override bool CanConvert(Type typeToConvert)
    {
        return typeToConvert.IsEnum;
    }

    public override JsonConverter? CreateConverter(Type typeToConvert, JsonSerializerOptions options)
    {
        var type = typeof(JsonStringEnumConverter<>).MakeGenericType(typeToConvert);
        return (JsonConverter)Activator.CreateInstance(type)!;
    }
}

public class JsonStringEnumConverter<TEnum> : JsonConverter<TEnum> where TEnum : struct, Enum
{

    private readonly Dictionary<TEnum, string> _enumToString = new();
    private readonly Dictionary<string, TEnum> _stringToEnum = new();
    private readonly Dictionary<int, TEnum> _numberToEnum = new();

    public JsonStringEnumConverter()
    {
        var type = typeof(TEnum);
        foreach (var value in Enum.GetValues<TEnum>())
        {
            var enumMember = type.GetMember(value.ToString())[0];
            var attr = enumMember.GetCustomAttributes(typeof(JsonPropertyNameAttribute), false)
                .Cast<JsonPropertyNameAttribute>()
                .FirstOrDefault();

            var num = Convert.ToInt32(type.GetField("value__")?.GetValue(value));
            if (attr?.Name != null)
            {
                _enumToString.Add(value, attr.Name);
                _stringToEnum.Add(attr.Name, value);
                _numberToEnum.Add(num, value);
            }
            else
            {
                _enumToString.Add(value, value.ToString());
                _stringToEnum.Add(value.ToString(), value);
                _numberToEnum.Add(num, value);
            }
        }
    }

    public override TEnum Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
    {
        var type = reader.TokenType;
        if (type == JsonTokenType.String)
        {
            var stringValue = reader.GetString();

            if (stringValue != null && _stringToEnum.TryGetValue(stringValue, out var enumValue))
            {
                return enumValue;
            }
        }
        else if (type == JsonTokenType.Number)
        {
            var numValue = reader.GetInt32();
            _numberToEnum.TryGetValue(numValue, out var enumValue);
            return enumValue;
        }

        return default;
    }

    public override void Write(Utf8JsonWriter writer, TEnum value, JsonSerializerOptions options)
    {
        writer.WriteStringValue(_enumToString[value]);
    }
}

Demo

var test = new Test()
{
    TestEnum0 = TestEnum.Enum0,
    TestEnum1 = TestEnum.Enum1,
    TestEnum2 = TestEnum.Enum2,
    TestEnum3 = TestEnum.Enum3,
};
Console.WriteLine(test);
Console.WriteLine("———————————————————————————————————————————————————————————————————");

var json = JsonSerializer.Serialize(test);
var obj = JsonSerializer.Deserialize<Test>(json);
Console.WriteLine(json);
Console.WriteLine(obj);
Console.WriteLine("———————————————————————————————————————————————————————————————————");

var str = """
    {
        "TestEnum0":"name1",
        "TestEnum1":0,
        "TestEnum2":3,
        "TestEnum3":2
    }
    """;
obj = JsonSerializer.Deserialize<Test>(str);
Console.WriteLine(obj);

record Test
{
    [JsonConverter(typeof(JsonStringEnumConverter))]
    public required TestEnum TestEnum0 { get; init; }

    [JsonConverter(typeof(JsonStringEnumConverter))]
    public required TestEnum TestEnum1 { get; init; }

    public required TestEnum TestEnum2 { get; init; }

    public required TestEnum TestEnum3 { get; init; }
}

enum TestEnum
{
    [JsonPropertyName("name0")]
    Enum0,

    [JsonPropertyName("name1")]
    Enum1,

    Enum2,

    Enum3,
}

Result

Test { TestEnum0 = Enum0, TestEnum1 = Enum1, TestEnum2 = Enum2, TestEnum3 = Enum3 }
———————————————————————————————————————————————————————————————————
{"TestEnum0":"name0","TestEnum1":"name1","TestEnum2":2,"TestEnum3":3}
Test { TestEnum0 = Enum0, TestEnum1 = Enum1, TestEnum2 = Enum2, TestEnum3 = Enum3 }
———————————————————————————————————————————————————————————————————
Test { TestEnum0 = Enum1, TestEnum1 = Enum0, TestEnum2 = Enum3, TestEnum3 = Enum2 }
bunnyi116 commented 1 year ago

Looking at the System.Text.Json.Serialization namespace, in addition to the JsonConverter and JsonConverter<T>, the only other JsonConverter added was the JsonStringEnumConverter. In other words, it was clearly identified that there is a frequent use case for converting between json string values and their enum equivalents, and vice-versa. However, right from the very beginning, people ran into situations where the string value is not necessarily the exact name of the enum, as it has been clearly demonstrated on stackoverflow: https://stackoverflow.com/questions/59059989/system-text-json-how-do-i-specify-a-custom-name-for-an-enum-value

I do agree that EnumMember is not the correct attribute, and that moving forward the goal should be to build upon System.Text.Json and not another unrelated namespace.

In the spirit of what's already been established with JsonStringEnumConverter, I would propose the following attribute:

namespace System.Text.Json.Serialization
{
    /// <summary>
    /// Specifies the enum string value that is present in the JSON when serializing and deserializing.
    /// </summary>
    [AttributeUsage(AttributeTargets.Field, Inherited = false, AllowMultiple = false)]
    public sealed class JsonStringEnumValueAttribute : JsonAttribute
    {
        /// <summary>
        /// Initializes a new instance of <see cref="JsonStringEnumValueAttribute"/> with the specified string value.
        /// </summary>
        /// <param name="value">The string value of the enum.</param>
        public JsonStringEnumValueAttribute(string value)
        {
            Value = value;
        }

        /// <summary>
        /// The string value of the enum.
        /// </summary>
        public string Value { get; }
    }
}

Which could then be used by the JsonStringEnumConverter specifically. Extending support for this attribute would eliminate the need for workarounds and nuget packages that people currently use and allow JsonStringEnumConverter to be used as the built-in go-to for all string to enum use cases.

I think it's right, a new name starting with Json should be used. According to the naming of JsonPropertyName, I think it might be more appropriate to name it JsonEnumNameAttribute or JsonEnumValueAttribute?

Corbie-42 commented 1 year ago

I think it's right, a new name starting with Json should be used. According to the naming of JsonPropertyName, I think it might be more appropriate to name it JsonEnumNameAttribute or JsonEnumValueAttribute?

I vote for JsonEnumNameAttribute, to that it is equal to the JsonPropertyNameAttribute. The value of the enum should not be serialized.

eduherminio commented 1 year ago

Am I right to understand that there's no built-in way to use System.Text.Json to deserialize an enum that comes as a string with dashes (i.e. "enum-1", "enum-2" values) to a C# enum?

KSemenenko commented 1 year ago

this is usful, can't wait for it

jscarle commented 1 year ago

The lack of built-in support in System.Text.Json (and .NET in general) for serializing and deserializing between arbritary string values and their enum representations is frustrating. This has been a gripe for years and it's always sort of just been pushed into a corner and forgotten about.

One of the solutions we've been actively using for a long time in our projects is the following approach.

First on any given enum, we define the string representation of the enum value as follows:

public enum ContactType
{
    [EnumMember(Value = "per")]
    Person,

    [EnumMember(Value = "bus")]
    Business
}

Then we use the extension methods defined below, we're able to convert from a string to the enum, and vice-versa. This then allows use to use the same mechanics in many different places such as in a System.Text.Json JsonConverter, in an Entity Framework ValueConverter<T1, T2>, a DTO mapper, or a LINQ query using a simple syntax:

// Converts "bus" to ContactType.Business.
var asEnum = "bus".FromEnumString<ContactType>();

// Converts ContactType.Person to "per".
var asString = ContactType.Person.ToEnumString();

Extension methods:

using System.Diagnostics;
using System.Runtime.Serialization;

public static class EnumExtensions
{
    public static string ToEnumString<TField>(this TField field)
        where TField : Enum
    {
        var fieldInfo = typeof(TField).GetField(field.ToString());
        if (fieldInfo is null)
            throw new UnreachableException($"Field {nameof(field)} was not found.");

        var attributes = (EnumMemberAttribute[])fieldInfo.GetCustomAttributes(typeof(EnumMemberAttribute), false);
        if (attributes.Length == 0)
            throw new NotImplementedException($"The field has not been annotated with a {nameof(EnumMemberAttribute)}.");

        var value = attributes[0]
            .Value;
        if (value is null)
            throw new NotImplementedException($"{nameof(EnumMemberAttribute)}.{nameof(EnumMemberAttribute.Value)} has not been set for this field.");

        return value;
    }

    public static TField FromEnumString<TField>(this string str)
        where TField : Enum
    {
        var fields = typeof(TField).GetFields();
        foreach (var field in fields)
        {
            var attributes = (EnumMemberAttribute[])field.GetCustomAttributes(typeof(EnumMemberAttribute), false);
            if (attributes.Length == 0)
                continue;

            var value = attributes[0]
                .Value;
            if (value is null)
                throw new NotImplementedException($"{nameof(EnumMemberAttribute)}.{nameof(EnumMemberAttribute.Value)} has not been set for the field {field.Name}.");

            if (string.Equals(value, str, StringComparison.OrdinalIgnoreCase))
                return (TField)Enum.Parse(typeof(TField), field.Name) ?? throw new ArgumentNullException(field.Name);
        }

        throw new InvalidOperationException($"'{str}' was not found in enum {typeof(TField).Name}.");
    }
}
gregsdennis commented 1 year ago

Not sure why I haven't already commented on this issue, but I've already published this in an extension library: Json.More.Net (docs).

dotMorten commented 1 year ago

I was sort of shocked that this isn't supported yet. The proposed workaround with custom contract resolvers are as so far as I can tell all runtime/reflection-based, and thus doesn't support AoT. Pretty much anything else in the json serialization can be adjusted to match your object model to the json, but enums are for some odd reason the exception. Why?

eiriktsarpalis commented 1 year ago

The proposed workaround with custom contract resolvers are as so far as I can tell all runtime/reflection-based, and thus doesn't support AoT.

Not all reflection is unsupported in Native AOT, for example this .NET 8 workaround works just fine:

using System.Diagnostics.CodeAnalysis;
using System.Reflection;
using System.Runtime.Serialization;
using System.Text.Json;
using System.Text.Json.Serialization;

MyPoco? result = JsonSerializer.Deserialize("""{"Value":"customName"}""", MyContext.Default.MyPoco);
Console.WriteLine(result!.Value); // Value

[JsonSerializable(typeof(MyPoco))]
public partial class MyContext : JsonSerializerContext
{
}

public class MyPoco
{
    [JsonConverter(typeof(JsonStringEnumConverterWithEnumMemberAttrSupport<MyEnum>))]
    public MyEnum Value { get; set; }
}

public enum MyEnum
{
    [EnumMember(Value = "customName")]
    Value = 42,
}

public sealed class JsonStringEnumConverterWithEnumMemberAttrSupport<[DynamicallyAccessedMembers(DynamicallyAccessedMemberTypes.PublicFields)] TEnum> 
    : JsonStringEnumConverter<TEnum> where TEnum : struct, Enum
{
    public JsonStringEnumConverterWithEnumMemberAttrSupport() : base(namingPolicy: ResolveNamingPolicy())
    {
    }

    private static JsonNamingPolicy? ResolveNamingPolicy()
    {
        var map = typeof(TEnum).GetFields(BindingFlags.Public | BindingFlags.Static)
            .Select(f => (f.Name, AttributeName: f.GetCustomAttribute<EnumMemberAttribute>()?.Value))
            .Where(pair => pair.AttributeName != null)
            .ToDictionary();

        return map.Count > 0 ? new EnumMemberNamingPolicy(map!) : null;
    }

    private sealed class EnumMemberNamingPolicy(Dictionary<string, string> map) : JsonNamingPolicy
    {
        public override string ConvertName(string name)
            => map.TryGetValue(name, out string? newName) ? newName : name;
    }
}
Uladzimir-Lashkevich commented 1 year ago

So, why such simple functionality cannot be provided out of the box (at least for new 's.t.j.-oriented' attribute, like JsonEnumNameAttribute) ?

rcollette commented 1 year ago

I'm surprised that no-one has pointed out that without a standard attribute for defining enum JSON values, OpenApi (Swashbuckle and Microsoft's own OpenApi discovery api) has no standard way of rendering the acceptable enumeration values in a way that isn't "custom". The work involved is more than just writing a JsonConverter implementation.

I'd like to have an enum of languages from rfc 7231 which include dashes in them, and have those enum values present in my OpenApi specification.

KSemenenko commented 1 year ago

Maybe in .net9, we can wait

jamiehankins commented 10 months ago

I'm dealing with a service that has an enum value that starts with a number. That obviously isn't legal in C#, so I've had to prepend an underscore to the value. This doesn't work for the service, so applying an attribute is the answer. It's frustrating to have to include a big custom converter class just to handle this one case when supplying an attribute to override the enum converter should be provided by default.

Previously, I had to implement my own snake-case converter. Switching to the default one now that I'm in .NET 8 is what sent me down this dark path.

gregsdennis commented 10 months ago

@jamiehankins if you've created the enum type, you can use the lib I link to in this comment.

jscarle commented 9 months ago

@eiriktsarpalis Can we please get this issue addressed in .NET 9.0 using whatever mechanism you believe provides the least amount of friction while still aligning with the internal guidelines for the language?

Perhaps support for a JsonEnumNameAttribute?

douglasg14b commented 7 months ago

It shouldn't require a JSON-specific attribute IMHO. Non-JSON processes often make use of EnumMember attributes for stringlifying/parsing of enum values. Having a JSON-specific attribute means we now need to double up our attribute usage.

Honestly there should be general framework support for such attributes and "string types".

jscarle commented 7 months ago

It shouldn't require a JSON-specific attribute IMHO. Non-JSON processes often make use of EnumMember attributes for stringlifying/parsing of enum values. Having a JSON-specific attribute means we now need to double up our attribute usage.

Honestly there should be general framework support for such attributes and "string types".

At this point, I'd take any support for any attribute.

KSemenenko commented 7 months ago

I can’t drop newtonsoft.json because of this issue

gregsdennis commented 7 months ago

@KSemenenko / @jscarle there is current support in Json.More.Net. You can move forward with that until .Net includes this natively. (Native AoT is also supported.)

https://docs.json-everything.net/more/examples/enums/

jscarle commented 7 months ago

@KSemenenko / @jscarle there is current support in Json.More.Net. You can move forward with that until .Net includes this natively.

https://docs.json-everything.net/more/examples/enums/

Plenty of solutions have been proposed as alternatives within this issue, all of which work fine. Finding a workaround is not the point. This absolutely should be implemented natively by the SDK as this is an extremely frequent use case, as demonstrated by both Newtonsoft.Json as well as the number of participants and links to this issue.

jscarle commented 7 months ago

Interim programme

Temporarily available methods

As a follow up to @bunnyi116's version of the JsonEnumConverter, I adjusted it with a few tweaks:

using System.Text;
using System.Text.Json;
using System.Text.Json.Serialization;

public class JsonStringEnumConverter : JsonConverterFactory
{
    public override bool CanConvert(Type typeToConvert)
    {
        return typeToConvert.IsEnum;
    }

    public override JsonConverter CreateConverter(Type typeToConvert, JsonSerializerOptions options)
    {
        var type = typeof(JsonStringEnumConverter<>).MakeGenericType(typeToConvert);
        return (JsonConverter)Activator.CreateInstance(type, options)!;
    }
}

public class JsonStringEnumConverter<TEnum> : JsonConverter<TEnum>
    where TEnum : struct, Enum
{
    private readonly Dictionary<TEnum, string> _enumToString = new();
    private readonly Dictionary<int, TEnum> _numberToEnum = new();
    private readonly Dictionary<string, TEnum> _stringToEnum = new(StringComparer.InvariantCultureIgnoreCase);

    public JsonStringEnumConverter(JsonSerializerOptions options)
    {
        var type = typeof(TEnum);
        var names = Enum.GetNames<TEnum>();
        var values = Enum.GetValues<TEnum>();
        var underlying = Enum.GetValuesAsUnderlyingType<TEnum>().Cast<int>().ToArray();
        for (var index = 0; index < names.Length; index++)
        {
            var name = names[index];
            var value = values[index];
            var underlyingValue = underlying[index];

            var attribute = type.GetMember(name)[0]
                .GetCustomAttributes(typeof(JsonPropertyNameAttribute), false)
                .Cast<JsonPropertyNameAttribute>()
                .FirstOrDefault();
            var stringValue = FormatName(attribute?.Name ?? name, options);

            _enumToString.Add(value, stringValue);
            _stringToEnum.Add(stringValue, value);
            _numberToEnum.Add(underlyingValue, value);
        }
    }

    private static string FormatName(string name, JsonSerializerOptions options)
    {
        return options.PropertyNamingPolicy?.ConvertName(name) ?? name;
    }

    public override TEnum Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
    {
        switch (reader.TokenType)
        {
            case JsonTokenType.String:
            {
                var stringValue = reader.GetString();

                if (stringValue is not null && _stringToEnum.TryGetValue(stringValue, out var enumValue))
                    return enumValue;
                break;
            }
            case JsonTokenType.Number:
            {
                if (reader.TryGetInt32(out var numValue) && _numberToEnum.TryGetValue(numValue, out var enumValue))
                    return enumValue;
                break;
            }
        }

        throw new JsonException($"The JSON value '{
                Encoding.UTF8.GetString(reader.ValueSpan)
                }' could not be converted to {typeof(TEnum).FullName}. BytePosition: {reader.BytesConsumed}.");
    }

    public override void Write(Utf8JsonWriter writer, TEnum value, JsonSerializerOptions options)
    {
        writer.WriteStringValue(_enumToString[value]);
    }
}

For those of you who are using minimal APIs, here's a quick tip. Add this to your services to add this converter to the default options used when serializing and deserializing requests:

using Microsoft.AspNetCore.Http.Json;

services.Configure<JsonOptions>(options => {
    options.SerializerOptions.Converters.Add(new JsonStringEnumConverter());
});

Note: This converter works under the assumption that the enum's underlying type is language's default of an int. If you use enums where the underlying type has been changed, this will explode.

eiriktsarpalis commented 7 months ago

@jscarle why write a converter from scratch when you can just piggyback on the naming policy as recommended in https://github.com/dotnet/runtime/issues/74385#issuecomment-1705083109? The built-in implementation works with Native AOT, handles non-int enums and Flags enums.

Assuming you want a non-generic factory (which won't work with AOT) you can extend my previous example as follows:

[RequiresUnreferencedCode("EnumMemberAttribute annotations might get trimmed.")]
[RequiresDynamicCode("Requires dynamic code generation.")]
public sealed class JsonStringEnumConverterWithEnumMemberAttrSupport : JsonConverterFactory
{
    public override bool CanConvert(Type typeToConvert) => typeToConvert.IsEnum;

    public override JsonConverter? CreateConverter(Type typeToConvert, JsonSerializerOptions options)
    {
        Type typedFactory = typeof(JsonStringEnumConverterWithEnumMemberAttrSupport<>).MakeGenericType(typeToConvert);
        var innerFactory = (JsonConverterFactory)Activator.CreateInstance(typedFactory)!;
        return innerFactory.CreateConverter(typeToConvert, options);
    }
}

Which can be used as usual:

JsonSerializer.Serialize(new { X = Foo.A | Foo.B }); // {"X":"First, Second"}

[Flags, JsonConverter(typeof(JsonStringEnumConverterWithEnumMemberAttrSupport))]
public enum Foo : ulong
{ 
    [EnumMember(Value = "First")] 
    A = 1uL << 60, 

    [EnumMember(Value = "Second")] 
    B = 2uL << 61
}
jscarle commented 7 months ago

@jscarle why write a converter from scratch when you can just piggyback on the naming policy as recommended in #74385 (comment)? The built-in implementation works with Native AOT, handles non-int enums and Flags enums.

Hello @eiriktsarpalis! Based off of your recommendation and your code examples, I created the following repository to compare a custom implementation versus extending the JsonStringEnumConverter.

I was able to solve 5 out of my 8 use cases using the AOT compatible source generated implementation you described. The three I was not able to solve are the following:

The first two use cases are nice to haves, but obviously not required. Perhaps you have some recommendations on how that could be solved using source generation.

However, that last one is a show stopper for me. The default implementation of the JsonStringEnumConverter does not validate the validity of the deserialized enum. It simply sets the backing field of the property.

image

Which means that any enum that is deserialized using the default implementation of the JsonStringEnumConverter cannot be trusted. Allowing the deserializer to deserialize data into an invalid state risks both the stability and the security of an application. This requires additional scrutiny of any deserialized enums which in turn leads to additional overhead caused by unnecessary validations. In the end, this simply causes the avoidance of the default implementation of JsonStringEnumConverter all together. Perhaps there's an issue that's already open for this?

I'd love to hear any feedback or recommendations on this. Thanks!

BoasHoeven commented 7 months ago

Kinda crazy that such basic features are still missing.

MSACATS commented 6 months ago

Honestly there should be general framework support for such attributes and "string types".

Wasn't System.Runtime.Serialization exactly this? Sigh.

At this point in my career I've accepted that we're doomed to a Sisyphean cycle of

  1. project is introduced
  2. slowly build it up and add the missing features (cue the "Perfection." meme)
  3. rebuild it from scratch with the new technology of the year, with only 80% of the functionality of the previous version
  4. goto 2

We ended up just using string properties in the JSON object classes and doing the conversions ourselves. Lame but it works.

0xced commented 4 months ago

Literally every time I work with System.Text.Json I need to convert enums which don't have a one-to-one mapping with the content of the JSON property and the C# property name.

I either write my own converters (tedious) or use the third party Macross.Json.Extensions NuGet package (not AOT compatible).

Like many others have already said in this thread, this feature really ought to be built-in to System.Text.Json. (This is currently the 3rd upvoted issue under area-System.Text.Json)

eiriktsarpalis commented 4 months ago

I either write my own converters (tedious) or use the third party Macross.Json.Extensions NuGet package (not AOT compatible).

Have you considered the workaround proposed here? https://github.com/dotnet/runtime/issues/74385#issuecomment-1705083109

eiriktsarpalis commented 4 months ago

To set expectations straight, System.Text.Json won't be supporting EnumMemberAttribute out of the box because the STJ assembly doesn't have a dependency on System.Runtime.Serialization. For those wanting to add EnumMemberAttribute support you can find suggested workarounds here and here.

API Proposal

Concerning the question of adding built-in enum name customization support, this would need to be done via a new attribute type:

namespace System.Text.Json.Serialization;

[AttributeUsage(AttributeTargets.Enum, AllowMultiple = false)]
public class JsonStringEnumMemberNameAttribute : Attribute
{
    public JsonStringEnumMemberNameAttribute(string name);
    public string Name { get; }
}

API Usage

Setting the attribute on individual enum members can customize their name

JsonSerializer.Serialize(MyEnum.Value1 | MyEnum.Value2); // "A, B"

[Flags, JsonConverter(typeof(JsonStringEnumConverter))]
public enum MyEnum
{
    [JsonStringEnumMemberName("A")]
    Value1 = 1,

    [JsonStringEnumMemberName("B")]
    Value2 = 2,
}
rcollette commented 4 months ago

@eiriktsarpalis That API Proposal seems appropriate. Perhaps it is implied that this will include alignment with API discovery such that string enumeration values are included (ex. for OpenAPI spec generation) but it wouldn't hurt to call that out as well.

julealgon commented 4 months ago

Have you considered perhaps an even more general version of the [DataMember] attributes, something that would sit on the DataAnnotations namespace along with stuff like [Display], or maybe in an even lower System level namespace/assembly, that would serve to indicate a more "raw" name override for the property/element/etc?

I feel like every library keeps redefining these attributes all the time with the same underlying purpose, and that we should instead come up with a more generic "renaming" mechanism that works more seamlessly with all libraries.

For example, what if there was a lower level attribute that actually changed the results of reflection, so that consumers would transparently honor the renamed fields/members even without actively searching for a specific attribute?

Such attribute could be used for basically every member type... including even methods. Then, when calls are made to either match or get those elements, the name override would be used/returned instead of the name defined in the identifier.

Example:

public class MyClass
{
    [MemberName("SomethingElseAltogether")]
    public void MyMethod()...
}

Then the calls would work like this:

var myMethodOriginal = typeof(MyClass).GetMethod("MyMethod"); // returns `null`. no match
var myMethodOverriden  = typeof(MyClass).GetMethod("SomethingElseAltogether"); // returns the member info, finds the match

Similarly, nameof(MyMethod) would return "SomethingElseAltogether", so this would be honored at compile-time as well.

This lower-level override would then propagate to every single consumer be it a source generator, a reflection-based scan or anything else, and honor the new names. Libraries would be simpler (as they don't have to check for attributes, or create custom ones) and the behavior would be finally unified.

Thoughts?

MSACATS commented 4 months ago

Have you considered perhaps an even more general version of the [DataMember] attributes, something that would sit on the DataAnnotations namespace along with stuff like [Display], or maybe in an even lower System level namespace/assembly, that would serve to indicate a more "raw" name override for the property/element/etc?

I feel like every library keeps redefining these attributes all the time with the same underlying purpose, and that we should instead come up with a more generic "renaming" mechanism that works more seamlessly with all libraries.

For example, what if there was a lower level attribute that actually changed the results of reflection, so that consumers would transparently honor the renamed fields/members even without actively searching for a specific attribute?

Such attribute could be used for basically every member type... including even methods. Then, when calls are made to either match or get those elements, the name override would be used/returned instead of the name defined in the identifier.

Example:

public class MyClass
{
    [MemberName("SomethingElseAltogether")]
    public void MyMethod()...
}

Then the calls would work like this:

var myMethodOriginal = typeof(MyClass).GetMethod("MyMethod"); // returns `null`. no match
var myMethodOverriden  = typeof(MyClass).GetMethod("SomethingElseAltogether"); // returns the member info, finds the match

Similarly, nameof(MyMethod) would return "SomethingElseAltogether", so this would be honored at compile-time as well.

This lower-level override would then propagate to every single consumer be it a source generator, a reflection-based scan or anything else, and honor the new names. Libraries would be simpler (as they don't have to check for attributes, or create custom ones) and the behavior would be finally unified.

Thoughts?

There is (was) a rich ecosystem for this sort of thing as it pertains to reflection (TypeDescriptor): see e.g. https://putridparrot.com/blog/dynamically-extending-an-objects-properties-using-typedescriptor/

I don't think the very low-level things (nameof/etc) are appropriate to change in this way -- if the name needs to be changed at such a low level, the motivation eludes me as to why you wouldn't change the actual name

jscarle commented 4 months ago

API Proposal

Concerning the question of adding built-in enum name customization support, this would need to be done via a new attribute type:

namespace System.Text.Json.Serialization;

[AttributeUsage(AttributeTargets.Enum, AllowMultiple = false)]
public class JsonStringEnumMemberNameAttribute : Attribute
{
    public JsonStringEnumMemberNameAttribute(string name);
    public string Name { get; }
}

API Usage

Setting the attribute on individual enum members can customize their name

JsonSerializer.Serialize(MyEnum.Value1 | MyEnum.Value2); // "A, B"

[Flags, JsonConverter(typeof(JsonStringEnumConverter))]
public enum MyEnum
{
    [JsonStringEnumMemberName("A")]
    Value1 = 1,

    [JsonStringEnumMemberName("B")]
    Value2 = 2,
}

That's perfect! 👍🏻

eiriktsarpalis commented 3 months ago

I spent a bit of time prototyping an implementation, but it turns out that the current proposed design stumbles on the source generator. TL;DR it would require resolving enum member names using reflection which in turn forces viral DynamicallyAccessedMembers declarations across a number of public APIs.

As it stands the proposed API isn't fit for purpose -- it would additionally require a number of extensions on the contract APIs such that custom enum metadata can be mapped at compile time by the source generator. This is nontrivial work, so it's possible that it won't make .NET 9 (for which feature development is set to conclude in the coming weeks). In the meantime I invite you to apply the workarounds as proposed here and here -- they provide a fully functional substitute that works with the existing EnumMemberAttribute.

API Proposal (Updated)

namespace System.Text.Json.Serialization;

+[AttributeUsage(AttributeTargets.Enum, AllowMultiple = false)]
+public class JsonStringEnumMemberNameAttribute : Attribute
+{
+    public JsonStringEnumMemberNameAttribute(string name);
+    public string Name { get; }
+}

namespace System.Text.Json.Serialization.Metadata;

public enum JsonTypeInfoKind
{
    None,
    Object,
    Enumerable,
    Dictionary,
+   Enum // Likely breaking change (kind for enums currently reported as 'None')
}

+[Flags]
+public enum JsonEnumConverterFlags
+{
+    None = 0,
+    AllowNumbers = 1,
+    AllowStrings = 2,
+}

public partial class JsonTypeInfo
{
    public JsonTypeInfoKind Kind { get; }
+   public JsonEnumConverterFlags EnumConverterFlags { get; set; }
+  // The source generator will incorporate JsonStringEnumMemberNameAttribute support by implementing its own naming policy
+  public JsonNamingPolicy? EnumNamingPolicy { get; set; }
}

Open Questions

The design needs to account for JsonStringEnumConverter annotations made on individual properties. The above only works for type-level annotations.

eiriktsarpalis commented 3 months ago

I've created a gist that combines both workarounds into one source file.

HavenDV commented 3 months ago

I'm creating an SDKs generator based on OpenAPI, and I also recently solved an issue with System.Text.Json and enums. I have a question - now for each enum I generate fast conversion extensions like these (https://github.com/andrewlock/NetEscapades.EnumGenerators), and for each enum I create my own converter that uses these extensions. This is fully compatible with Trimming/NativeAOT and logically should also be performant, but I am confused by hundreds and thousands of converters for complex APIs, can their presence negate the advantages?

bunnyi116 commented 3 months ago

I think I think JsonPropertyName can be NativeAOT, and I think it can also achieve this function.

eiriktsarpalis commented 3 months ago

After further experimenation, I was able to produce an implementation that makes the attribute work in AOT without added APIs. Now to API review this.

eiriktsarpalis commented 3 months ago

API as proposed in https://github.com/dotnet/runtime/issues/74385#issuecomment-2220667024 has been approved over email. cc @stephentoub @terrajobst

jscarle commented 3 months ago

@eiriktsarpalis Thank you for completing this! I have a couple of questions?

1) Will this be part of .NET 9?

2) Is deserializing case insensitive?

3) Does deserializing now throw an exception on invalid values?

eiriktsarpalis commented 3 months ago
  1. Yes, it should be available in Preview 7.
  2. No. This is similar to how JsonNamingPolicy is being handled by the converter.
  3. Yes, if you are asking about invalid string identifiers. Numeric values would still be picked up if the converter has been configured accordingly.