Allowing customization of JSON key ordering when serializing dictionaries

rummelsworth commented 4 years ago

Source type

Dictionary<string, int> // for example, with ("a", 1), ("c", 2), and ("b", 3), added in that order.

Desired JSON

{"a":1,"b":3,"c":2}

Actual JSON

{"a":1,"c":2,"b":3}

Steps to reproduce

void Main() // in linqpad
{
    var d = new Dictionary<string, int>
    {
        ["a"] = 1,
        ["c"] = 2,
        ["b"] = 3,
    };
    var s = new JsonSerializerSettings
    {
        ContractResolver = new OrderedPropertyNamesContractResolver(StringComparer.OrdinalIgnoreCase),
    };
    JsonConvert.SerializeObject(d).Dump(); // yields {"a":1,"c":2,"b":3}, as expected.
    JsonConvert.SerializeObject(d, s).Dump(); // yields the same, but was hoping for {"a":1,"b":3,"c":2}
}

// My very naive attempt at doing this without constructing a whole new SortedDictionary or SortedList ---
class OrderedPropertyNamesContractResolver : DefaultContractResolver
{
    readonly IComparer<string> Comparer;

    public OrderedPropertyNamesContractResolver(IComparer<string> comparer) => Comparer = comparer ?? throw new ArgumentNullException(nameof(comparer));

    protected override IList<JsonProperty> CreateProperties(Type type, MemberSerialization memberSerialization)
    {
        var properties = base.CreateProperties(type, memberSerialization).OrderBy(property => property.PropertyName, Comparer).ToList();
        foreach (var (property, index) in properties.Select(ValueTuple.Create<JsonProperty, int>))
        {
            property.Order = index;
        }
        return properties;
    }
}

Current workaround

Construct a SortedDictionary or SortedList with the same entries as the source dictionary, and then serialize that. Possibly insufficient when the dictionary is actually inside an aggregate that you're serializing.

bartelink commented 4 years ago

Dictionaries are by definition arbitrarily ordered - a special case like this makes no sense to bake into a general library

rummelsworth commented 4 years ago

[Dictionary keys] are by definition arbitrarily ordered

Agreed. So are POCO properties. Yet it makes lots of sense that custom ordering of the latter is allowed via JsonPropertyAttribute.Order and (e.g. within a custom contract resolver) JsonProperty.Order.

a special case like this makes no sense to bake into a general library

Elaboration on this point would be appreciated.

bartelink commented 4 years ago

It's not a thing in the json spec.

Json.Net is a library thats ~10 years old. You have a workaround for your special case. It's pretty difficult to justify adding this case to library thats this mature and has survived this long without the special casing at this point.

But: once again: if a low level json writer was arbitrarily reordering items, that would be a problem. This is a Dictionary - I'd be very shocked and surprised that there's a way to order them.

TL;DR This is a "starts with -100 points" issue

rummelsworth commented 4 years ago

TL;DR This is a "starts with -100 points" issue

The implication being that some issues do not start with -100 points. Contrast this with what you linked:

So, we decided on the additive approach instead, and worked hard to keep the complexity down. One way to do that is through the concept of “minus 100 points”. Every feature starts out in the hole by 100 points, which means that it has to have a significant net positive effect on the overall package.

So again, I think we agree. This feature, like all features, starts with -100 points.

It's not a thing in the json spec.

Correct, I think. Not sure how this is relevant, though, since JSON.net allows custom ordering of JSON object keys when the keys come from .NET object properties.

Json.Net is a library thats ~10 years old. You have a workaround for your special case. It's pretty difficult to justify adding this case to library thats this mature and has survived this long without the special casing at this point.

Again, mostly correct. It is about 14 years old. I have a partial workaround. The feature is indeed probably difficult to justify.

if a low level json writer was arbitrarily reordering items, that would be a problem.

So... it's a problem that the current API allows customization of JSON object key ordering for .NET object properties. I disagree here, but it's a perfectly valid opinion for you to hold.

This is a Dictionary - I'd be very shocked and surprised that there's a way to order them.

I'm sincerely trying to stand in your shoes on this, but I don't understand your "shock & surprise" here:

Things with comparers can be ordered.
Strings have comparers.
JSON object keys are strings.
JSON.net provides custom ordering when those keys are sourced from .NET object properties.
JSON.net does not provide custom ordering when those keys are sourced from .NET dictionary types.

To be clear, I make no claim that this feature is easy to (a) implement or (b) integrate without breaking changes. I hope that it is both of those things, but if wishes were horses, beggars would ride.

I'm removing the "please" from the issue title, but not because I'm not a beggar. I am a beggar, as is almost any FOSS user making any request of the FOSS project they're using. I think the "please" implies I'm unwilling to take this on myself. I am willing. If I manage to schedule some time for it, I'll mention it here, just as a heads-up for other potential implementors/collaborators.

bartelink commented 4 years ago

I was being a devil's advocate - I won't be implementing the feature. I like software I use to be and stay simple. I also like maintainers not to have to ask devils advocate questions that others could ask. I consider that a public service.

I don't think your feature is completely crazy but a) all features start with -100 b) Dictionaries, no matter how comparable their keys or vaales are not ordered c) this is not a systemic issue affecting lots of cases - its a non-idiomatic JSON thing that you have a workaround for

I'm not suggesting you're crazy or this is unimplementable, but having a general library support a very specific rendering like this smells funny to me.

That's all, no harm intended.

rummelsworth commented 4 years ago

(a) Agreed, as mentioned above.
(b) Agreed, but irrelevant as discussed above.
(c) Agreed, but the workaround is partial, as mentioned above.

No offense taken. I think it's great you engaged with this. At the very least, our discussion has been a "smoke test" of the feature's reasonableness. Thanks.

AlexSnowLeo commented 4 years ago

Hi @rummelsworth and @bartelink. You will be surprised, but I recently came across a similar problem. It is not concerned with sorting, but it stems from the fact that the basic method CreateProperties does not return properties for the type of dictionary. For example:

If I use Dictionary with determined value type 'object' - CreateProperties called, but the basic method does not return anything.
If I use Dictionary with determined value type 'int' - CreateProperties method is not called.

This is expected behavior?

robheffo79 commented 2 years ago

I personally would like to see this added. I am using Json serialisation to produce a "normalised" versions of arbitrary objects. Being able to ensure Dictionarys have consistantly ordered keys helps greatly because a dictionary doesn't care about the order, just that a key is present or not, whereas the normalisation process does care about both the order and whether or not a key is present..

danielschilling-ml commented 2 weeks ago

For reference - here is similar functionality in other languages...

JavaScript

const stringify = require('json-stable-stringify');
console.log(stringify({"c": 0, "b": 0, "a": 0}));

Python

json.dumps({"c": 0, "b": 0, "a": 0}, sort_keys=True)

JamesNK / Newtonsoft.Json