System.Text.Json support to System.Runtime.Serialization

[bruno-garcia]

This is a follow up to this comment:

Is there a plan to support [DataContract], [DataMember] and friends?

Today this is a way to define how types should serialize/deserialize (e.g. the field name) in a way that doesn't bring a dependency to any serialization library to the project using it.

The current JsonNamingPolicy takes a string only so there's no way to inspect the member's attributes. So it seems so far there's no way to support annotations (.NET Attributes) and as such no way to define the serialized name except from the original property name.

Motivation

System.Runtime.Serialization is part of .NET Standard and it allows us to annotate members of a class in a way to hint a serialization library what name to use and whether to include it or not if the value is null.

This is valuable because it allows us to have NuGet packages that don't depend on a serialization library directly.

Ensuring that a serialized version of a type matches a certain protocol could be done by means of tests only. This way we can make sure more than one library would be supported (like Newtonsoft.Json and DataContractSerializer at this point.

[ahsonkhan]

We don't currently have plans to support the attributes from System.Runtime.Serialization

[bruno-garcia]

@ahsonkhan Is there a plan to provide an API to define an arbitrary name for a serialized property? Not necessarily via attributes.

[dazinator]

Not supporting this is a fail for the .NET library ecosystem. It means nuget package authors that provide serialisable models in their libraries (with Blazor in the picture now, there are nuget packages that provide server side and client side code, and models that must be serialised in between) - must pass on their choice of serializer to their consumers - i.e System.Text.Json vs Newtonsoft.Json. This will result in blazor client applications being forced to consume both dependencies (not ideal as blazor wasm applications need as few dll's as possible to avoid poor startup performance in the browser), or nuget package authors having to publish two versions of their solution, one for each serialiser - no one will do that.

I'd be fine if this wasn't implemented yet - but there were plans to. But the fact there are't even any plans to implement it makes me think this is an oversight. Why would you not plan to implement this?

[daerhiel]

We don't currently have plans to support the attributes from System.Runtime.Serialization

There will be no use of System.Text.Json then. I expect to be able to transfer the serialization model when migrating over versions, so I won't have to redesign existing app architecture.

[ahsonkhan]

Is there a plan to provide an API to define an arbitrary name for a serialized property? Not necessarily via attributes.

@bruno-garcia, you could try and register a converter for a particular type/property and override the behavior. I don't know how you can customize any arbitrary property name though without attributes. Do you have a a scenario in mind where you'd need that without attributes? @steveharter - do you have any thoughts on how we can support this? @bruno-garcia - it would be good if you could file a separate issue for this with your expectation/use case.

This will result in blazor client applications being forced to consume both dependencies (not ideal as blazor wasm applications need as few dll's as possible to avoid poor startup performance in the browser), or nuget package authors having to publish two versions of their solution, one for each serialiser - no one will do that.

That's a good point, and I agree that decoupling the dependency to a particular serializer is useful. I'll spend some time taking a look at this space in the near future.

I'd be fine if this wasn't implemented yet - but there were plans to. But the fact there are't even any plans to implement it makes me think this is an oversight.

There are a lot of feature/capability requests when it comes to serialization and we prioritized enabling key scenarios first and hence only certain features can be supported for the given release. To clarify, I am not suggesting we would never support this, just that we haven't gone through and spec'd out this particular feature (the requirements/constraints/etc.). Which is why I said currently and which is why the issue is still open. We are still doing 5.0 planning.

Why would you not plan to implement this?

Part of the reservation was concerns around binary serialization/supporting attributes like [OnSerializing] (which is something I need to get clarity on anyway, and as you mentioned S.R.Serialization has general purpose attributes too - https://github.com/dotnet/corefx/issues/37503#issuecomment-500969445). But it's primarily scheduling and filing the gaps for other features took precedence (see current backlog - https://github.com/dotnet/corefx/issues?q=is%3Aopen+is%3Aissue+label%3Aarea-System.Text.Json+milestone%3A5.0). Feedback here about usage/scenarios being blocked certainly helps motivate enabling this. I appreciate folks sharing their current usage patterns and current pain points of migrating.

I expect to be able to transfer the serialization model when migrating over versions, so I won't have to redesign existing app architecture.

Given the current release of the JSON stack has bare-bones feature-set with an emphasis on performance first, this isn't feasible for all cases already (for example if your model had fields instead of properties). Going from using Newtonsoft.Json where you may be relying on all the features it offers, you will certainly find gaps where migration would require changes (or it would be non-feasible) for the foreseeable future. I doubt we will ever have complete parity here, but we will focus on closing the most significant gaps in subsequent releases (especially where the new stack is better suited to solve the problem).

[bruno-garcia]

Do you have a a scenario in mind where you'd need that without attributes? I don't. Ideally we'd be able to use it as-is. I can use DataContractSerializer or Newtonsoft.Json with my types annotated with System.Runtime.Serialization and it works fine. When I raised the ticket the only thing the converter was able to do is to give the json name based on the property name (not the type's member instance, so that i could look for the attribute myself.

I understand the design here was focused on perf and doing reflection to look for the attribute would defeat the purpose. I'm accepting that fact we'll never get the support but I won't close the ticket myself. :)

[vitidev]

@bruno-garcia

I understand the design here was focused on perf and doing reflection to look for the attribute would defeat the purpose.

There is no difference between resolving of attribute [DataMember] or [JsonPropertyName]. This is doing once and does not affect the final performance.

[bruno-garcia]

@vitidev when I raised this, if I didn't get it wrong, no attribute was resolved. The only way to change the name of the property in the resulting JSON was by inspecting the object's property name (IIRC it took a string and returned a string). If instead it returned the Property object (reflection) we could inspect the attribute to look for the alt name. Ideally we could register some Func<> or type that would be called and upon registration we could do the reflection bit (once) and register some compiled expressions to do the conversion. So the reflection perf hit would happen only at startup time.

[layomia]

As far as supporting attributes from System.Runtime.Serialization is concerned, it is not on the System.Text.Json roadmap to do this en masse. Please open individual issues for each attribute/logical group of attributes for which support is needed, and include use cases.

[bpicolo]

@layomia Should #30009 be reopened, then? It was closed due to this ticket being open. A lot of useful libraries in the ecosystem depend on this for serialization to/from many formats, json included. That sort of compatibility seems to be what dotnet strives for.

If performance impact is a concern, could it potentially be factored into JsonSerializerOptions?

[layomia]

Reopening this issue based on the comments from @steveharter in https://github.com/dotnet/runtime/issues/30009#issuecomment-697936065:

Early-on during design of STJ in 3.0, it was decided not to support pre-existing attributes mainly because they would only be partially supported and it would be hit-and-miss meaning STJ would only support some of those in the first release and additional support added in future releases. This would cause endless confusion over what is and what is not supported. STJ can't just throw NotSupportedException for the unsupported cases since that would mean the attribute usage would need to be removed from the corresponding types (not feasible unless the types are owned), or have a way to turn off the exception.

Also since STJ would need to explicitly look for those attributes, it would cause some slowdown during warm-up.

Consider just the DataMemberAttribute properties:

• EmitDefaultValue: in 3.0, we couldn't support since we didn't have that feature. Now in 5.0 it is.
• IsNameSetExplicitly: not supported yet.
• IsRequired: not supported yet.
• Name: supported through STJ.PropertyNameAttribute.
• Order: not supported yet.

Then there's also a few other attributes including CollectionDataContractAttribute, OnDeserializedAttribute that would also contribute to hit-and-miss. STJ also didn't consider other areas including System.IConvertible.

So we went with a new explicit model that is intuitive (if it's not in the STJ namespace then it's not supported) that is high-performance with the thought we can add support for pre-existing attributes through an opt-in mode\flag of some sort or perhaps a pluggable metadata provider. These have not been implemented yet.

Moving-forward, having a pluggable metadata provider along with an opt-in System.Runtime.Serialization "compat" implementation of that provider makes sense to me. Not sure if that has enough priority for 6.0, but it could be considered along with the feature to expose metadata and more flexible converters.

[bruno-garcia]

we can add support for pre-existing attributes through an opt-in mode\flag of some sort or perhaps a pluggable metadata provider.

That's all we want, thanks.

Not sure if that has enough priority for 6.0, but it could be considered along with the feature to expose metadata and more flexible converters.

If it's not landing on 6.0, that means 2022 the earliest. I really hope I can unsubscribe from this issue before that. :) But I'm sure it'll be helpful to many folks that have large code bases that rely on these.

[LukeTOBrien]

This is a majour issue for me (and a supprise, what's the point of [DataMember] if they are not used in JSON?) as I am consuming a third-party REST API that is using underscore_naming - So the logical way to overcome this is to use [DataMember].

I will switch back to JSON.Net until this is implemented.

[ahsonkhan]

as I am consuming a third-party REST API that is using underscore_naming

If that is the only issue you are dealing with, you could implement a a snake case naming policy and set that in the JsonSerializerOptions, similar to how camel casing is set: https://docs.microsoft.com/en-us/dotnet/standard/serialization/system-text-json-customize-properties#use-camel-case-for-all-json-property-names https://github.com/dotnet/runtime/pull/42009

[xt0rted]

I added a snake case naming policy to one of my projects. It's using code copied over from json.net. Here's an example of it:

https://github.com/JamesNK/Newtonsoft.Json/blob/42139ea6cd8d500790bf27c9afc69fd66eab60ad/Src/Newtonsoft.Json/Utilities/StringUtils.cs#L206

public class JsonSnakeCaseNamingPolicy : JsonNamingPolicy
{
    public override string? ConvertName(string name) => StringUtilities.ToSnakeCase(name);
}

var mvcBuilder = services.AddControllersWithViews()
    .AddJsonOptions(options =>
    {
        options.JsonSerializerOptions.WriteIndented = Environment.IsDevelopment();
        options.JsonSerializerOptions.PropertyNamingPolicy = new JsonSnakeCaseNamingPolicy();
    })
    .AddFluentValidation();

[LukeTOBrien]

Thank you, I will try this. A decrotive attribue way of doing this would be more convienient thought.

If that is the only issue

I also have issues with List<Interface, but that;s another story

[HeathJared]

Any updates or ETA on this? We've been tracking this for 2 years and is the primary reason we have not switched from multiple third party libraries (and .NET Core 3.1).

We HEAVILY use [DataMember(Name="")], [EnumMember], and [IgnoreDataMember] with multiple different serializers and formats.

We also use obfuscation which prevents relying on naming policies.

There is a zero percent chance we can use this library if we're forced to use [JsonPropertyName] and other "proprietary" attributes.

Based on More extensible object and collection converters #36785 being a part of Milestone 6.0.0 can we assume this is scheduled to release in November 2021 as part of .NET 6?

[Grauenwolf]

This is turning out to be a major problem for me as well. I am in the middle of a .NET Core 3.1 conversion and the lack of support for these attributes is causing hard to find bugs in my code.

[Grauenwolf]

@bruno-garcia

I understand the design here was focused on perf and doing reflection to look for the attribute would defeat the purpose.

There is no difference between resolving of attribute [DataMember] or [JsonPropertyName]. This is doing once and does not affect the final performance.

What I don't understand is why all of these new attributes were created in the first place. It's like the author didn't understand the .NET ecosystem.

Now I have to search through all of my code that was using DataContract/DataMember to control what data is revealed via REST and update it for this new set of attributes.

This is a huge security vulnerability for me that I only happened on by accident.

EDIT: Ok, I see why it was done. But I still think it was a huge mistake. We assume that not all properties of those attributes will be honored by every serializer. We don't assume that they'll be ignored completely.

[LukeTOBrien]

Yes, to me it doesn't make sense to create a separate attribute, why not use existing attribute?

On Thu, 17 Jun 2021, 17:29 Jonathan Allen, @.***> wrote:

@bruno-garcia https://github.com/bruno-garcia

I understand the design here was focused on perf and doing reflection to look for the attribute would defeat the purpose.

There is no difference between resolving of attribute [DataMember] or [JsonPropertyName]. This is doing once and does not affect the final performance.

What I don't understand is why all of these new attributes were created in the first place. It's like the author didn't understand the .NET ecosystem.

Now I have to search through all of my code that was using DataContract/ DataMember to control what data is revealed via REST and update it for this new set of attributes.

This is a huge security vulnerability for me that I only happened on by accident.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/dotnet/runtime/issues/29975#issuecomment-863386804, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD3KMFLBK3FB5RHNGSMAATLTTIPIHANCNFSM4MCQ65JA .

[layomia]

Triage - moving this issue to future. We aren't going to support these attributes natively in the serializer, but will work toward allowing users to easily detect them with minimal configuration code, and set the corresponding type metadata to influence (de)serialization - https://github.com/dotnet/runtime/issues/34456.

[davidnmbond]

We don't currently have plans to support the attributes from System.Runtime.Serialization

...making this library useless.

[LukeTOBrien]

It is not useless if there is a fluent API way of doing the same thing.
Nowaday I try not to use attribute because I want my entities to be a POCO class library with no dependencies .
IMO use should support attributes and fluent API as well.

There have also been a few other issues that have made me return to JSON.NET on the server side, one issue was an interface using a different converter depending on a enum value or depending on existence of a property.

[Grauenwolf]

Not useless, but it makes it incredibly difficult to use with legacy code.

I really don't understand why this can't be an opt-in setting.

[markusschaber]

Could that fluent api used to create helper / wrapper functions, maybe as an AddOn NuGet Package?

[eiriktsarpalis]

Tagging #29975, which proposed adding support for EnumMemberAttribute. If we do end up supporting for this attribute it should be recognized by JsonStringEnumConverter.

[hkusulja]

Why is this still open, and not acknowledged and resolved? Microsoft should merge and accept this? Code is simple in mentioned extension. When can we expect the progress and resolution on this?

[AroglDarthu]

@eiriktsarpalis Tagging #31081 as well. https://github.com/dotnet/runtime/issues/31081#issuecomment-635521221 goes beyond just System.Runtime.Serialization.

[hkusulja]

@layomia can you update status here? it should be simple to add this in current dotnet 6 ...

[nathando]

hi, can I second the request for status update on dotnet 6 please ?

[eiriktsarpalis]

.NET 6 has already been released and as such it currently only qualifies for servicing updates. Please see our .NET release, patches, and support document for more information.

[davidnmbond]

Stick with NewtonSoft everyone.

[hkusulja]

@eiriktsarpalis Your mentioned link does not say anything about .NET 6, only .NET 5 and older this thread is several years older than .NET 6 I believe this can be implemented with .NET 6.1 features, or at least promise and put it into .NET 7 roadmap.

[layomia]

This is being considered for .NET 7.0. The "Developers can customize the JSON serialization contracts of their types" user story (https://github.com/dotnet/runtime/issues/63686) would provide the underlying APIs to set the serialization metadata specified by System.Runtime.Serialization attributes. A simple resolver API (either inbox, user, or community-package authored) could then been written on top of this to parse the data from the attributes and register them with the serializer.

[knightmeister]

I want to +1 this. DataContract has been standard in .NET for a long time and is used extensively in my code base. While I can work around it, it’s a major pain in the proverbial and really shouldn’t be something I need to work around.

Did the original author not know .NET?

I really hope this is considered for the 7.0 timeframe.

[crozone]

@eiriktsarpalis Tagging #31081 as well. #31081 (comment) goes beyond just System.Runtime.Serialization.

31081 should not have been closed, it was replaced with this issue but this issue isn't going anywhere judging by this comment. Correct enum serialization is an essential feature of any serialization library. JamesNK flagged it as an essential feature pre core 3.0. The package that provides a workaround has hit 2.8 million downloads. It is a fundamental expectation of any competent JSON serialization library and the fact that there's still no built in implementation is honestly bewildering.

Now the lack of System.Runtime.Serialization support and the refusal to apparently even consider it as an option is disheartening. I don't mean to be rude but how is it possible that the team is this out of touch with the needs of its real-world customers? What is the goal of System.Net.Json? Obviously bleeding edge performance is an admirable goal but ultimately useless if nobody can utilize the library to actually do what they need it to do.

We don't need the entirety of System.Runtime.Serialization implemented and what would work well doesn't need to happen all at once, we just need the basic tools to cover the 99% of usecases we hit in the real world. That basically boils down to being able to control the basics of how fundamental types get serialized and what names they get given. The functionality could even be implemented in a separate package as a set of bolt-on custom serializers if System.Runtime.Serialization is really considered that much of a black sheep and separation of concerns is really that much of an issue.

Newtonsoft.Json has had all of this solved using the standard System.Runtime.Serialization attributes for over a decade and 2 billion package downloads later people are fairly happy with it. It is incredibly frustrating to see that these features apparently still need to have their existence justified with useless debate in github issues that go nowhere and get closed over and over again.

[eiriktsarpalis]

The package that provides a workaround has hit 2.8 million downloads. It is a fundamental expectation of any competent JSON serialization library and the fact that there's still no built in implementation is honestly bewildering.

And why is this a bad thing? Let me be absolutely clear, providing built-in support for everything is an explicit anti-goal for System.Text.Json. Instead, we aim to provide good extensibility points so that third-party extensions like the one you cited can be successful.

We have no plans on adding built-in support for System.Runtime.Serialization attributes, per the response in https://github.com/dotnet/runtime/issues/29975#issuecomment-1011584768 we expect that contract customization should let users opt in to any attribute that suits their use case.

[Grauenwolf]

I don't understand.

System.Text.Json is a built-in library. It's part of the SDK.

The System.Runtime.Serialization attributes are also a built-in library, shipped as part of the SDK.

Why is a Microsoft-written library not going to honor the Microsoft-defined standard for serialization libraries?

And how do you know #63686 is even right if you don't have anything inside the library that actually uses it?

[niemyjski]

Granted, I think the team should have supported DataMemberAttribute but I agree this shouldn't support the kitchen sink. It would have been nice if it supported System.Runtime.Serialization attributes possibly in a second package but supporting this now is a major breaking change as it would change the behavior of all .NET Core apps.

[Grauenwolf]

1. Why would be a breaking change? Can't it just be an feature we turn on with a setting?
2. If it's a breaking change when shipped within the library, wouldn't it also be a breaking change when shipped as a separate lirbary?

[eiriktsarpalis]

I don't understand.

System.Text.Json is a built-in library. It's part of the SDK.

The System.Runtime.Serialization attributes are also a built-in library, shipped as part of the SDK.

Why is a Microsoft-written library not going to honor the Microsoft-defined standard for serialization libraries?

The attributes in System.Runtime.Serialization have been specifically designed for serializers like BinaryFormatter and DataContractSerializer, and are not general-purpose serialization annotations. They might have been adopted by third-party serialization libraries out of convenience, but there is no "Microsoft-defined standard" for serialization libraries as such. The design of System.Text.Json has been very deliberate in defining its own sets of attributes, often duplicating what exists in System.Runtime.Serialization. And to be clear, just because two libraries are shipped as part of the shared framework, it doesn't mean that they have to interoperate.

That being said, we do recognize the need for some users to use System.Runtime.Serialization annotations in System.Text.Json. We want to offer extensibility points to make adding such support easy, but we currently have no plans on providing it out of the box.

And how do you know https://github.com/dotnet/runtime/issues/63686 is even right if you don't have anything inside the library that actually uses it?

That's what tests are for :-)

[Grauenwolf]

The attributes in System.Runtime.Serialization have been specifically designed for serializers like BinaryFormatter and DataContractSerializer,

That's not true. BinaryFormatter is a .NET 1 feature based around the ISerializable interface.

DataContractAttribute was introduced in .NET 3. It is described as "Specifies that the type defines or implements a data contract and is serializable by a serializer". It does on to say "DataContractSerializer" is an example of one such serializer, but it's not the only one. And there's a reason for that.

The whole concept of data contracts was to create a standard way of handling serialization. Classes would implement the contracts, and then the developer, as a separate decision, would choose which serialization library or libraries they wish to use.

[Grauenwolf]

And how do you know https://github.com/dotnet/runtime/issues/63686 is even right if you don't have anything inside the library that actually uses it?

That's what tests are for :-)

No. It doesn't work that way.

Your tests only prove that the code you wrote is the code that you thought you wrote. It doesn't actually prove that the code you wrote is suitable for purpose.

It can pass every test and still have a fundamental design flaw that makes it impossible to actually be used with data contracts. To prove that this feature is actually what we need, you have to actually implement something with it. And if you do the data contract implementation, you might as well put that implementation in the library rather than just the test cases.

[eiriktsarpalis]

The whole concept of data contracts was to create a standard way of handling serialization. Classes would implement the contracts, and then the developer, as a separate decision, would choose which serialization library or libraries they wish to use.

Perhaps, but like I said the designers of System.Text.Json made the conscious decision to deviate from what might have been considered standard 16 years ago. It's not exactly unheard of for older patterns and practices to be deemphasized in favor of newer iterations.

Your tests only prove that the code you wrote is the code that you thought you wrote. It doesn't actually prove that the code you wrote is suitable for purpose.

I think you might be making a point about testing in general. My acceptance tests for component Foo might be invalid or inadequate regardless of whether it's shipped in the shared framework or not.

[Grauenwolf]

Perhaps, but like I said the designers of System.Text.Json made the conscious decision to deviate from what might have been considered standard 16 years ago. It's not exactly unheard of for older patterns and practices to be deemphasized in favor of newer iterations.

Making a "conscious decision" does not mean it was the correct decision. Saying, "I was awake when I made this choice" doesn't justify the choice. It doesn't even explain it.

And the "new pattern" is not new. It's the same pattern with different attribute names.

[manigandham]

Let me be absolutely clear, providing built-in support for everything is an explicit anti-goal for System.Text.Json.

Nobody is asking to support everything.

The attributes in System.Runtime.Serialization have been specifically designed for serializers like BinaryFormatter and DataContractSerializer, and are not general-purpose serialization annotations. They might have been adopted by third-party serialization libraries out of convenience, but there is no "Microsoft-defined standard" for serialization libraries as such.

That might have been the intention at the start but the standard is what they've become over time. How the community uses it is what matters, not some arcane documentation. You know this, we know this, everyone already agrees on the unofficial but official standard.

That's why this adamant position is so strange given that the design is just new attributes that literally do the same thing and are used the same exact way, but offer nothing except incompatibility for so much existing code.

Why stand in the way of what the community overwhelmingly expects and wants? What's the fundamental gain here?

[mfcallahan]

We don't currently have plans to support the attributes from System.Runtime.Serialization

Based on the fact that this comment received 0 thumbs-up and well over 100 👎 thumbs down, it seems that the community really disagrees with that decision. What would it take to change that? This feels like two teams not collaborating and the end user suffering.

[dmcnaughton-nbc]

I agree with the overall sentiment here that support for System.Runtime.Serialization is an important feature that a lot of users would benefit from. This has been the defacto way of decorating data contracts within .NET for a long time and without this support it makes shifting from Newtonsoft to the new System.Text.Json much more of a burden for existing applications.

I'm not sure why there's such push back from this team regarding this feature, it's a fundamentally reasonable request that enables far better interoperability with existing codebases since a majority of people who have JSON data contracts are likely using Newtonsoft. I would understand if this was going to require an additional dependency outside of the .NET standard, but with System.Runtime.Serialization being part of .NET Standard. And like @bruno-garcia stated, this would not require additional external dependencies for people publishing libraries such as NuGet pacakges for general consumption.

[luetm]

After reading through this I can appreciate the position of the System.Text.Json team here. If they start adding things left right and center, the library will get bloated and more complicated every time. Extending it will be more difficult because of all the interdependencies, and performance will get worse. And then we complain again.

If you expect System.Runtime.Serialization attributes to work, a good extension apparently exists already. Who cares if it doesn't come from Microsoft, Newtonsoft.Json didn't either.

I get the feeling that some are expecting this to become Newtonsoft.Json 2.0. For me having System.Text.Json be the default, fast and simple approach I can extend, and Newtonsoft to be the full fledged, slower one that has everything built in is totally fine, as long as I can replace it easily.

[Grauenwolf]

If you expect System.Runtime.Serialization attributes to work, a good extension apparently exists already.

They say we may get the ability to write such an extension in .NET 7.

As far as I can tell, the linked library is unrelated to what we're talking about.