Closed terrajobst closed 4 years ago
This is great. I am all for faster and less allocating json parsing.
Will there be a discussion about the features from json.net that the new json apis will support? If there is,I think the two major features that come to mind would be renaming/casing properties and ignoring null properties.
Will there be a discussion about the features from json.net that the new json apis will support?
Yes. We've done some early thinking that we will migrate to CoreFx. It will be a feature that is designed & built in the open as usual. In addition, I've reached to authors of many of the popular JSON libraries and invited them to review early drafts of this announcement. My hope is that we can work together to create a solid JSON component for the platform while also keeping the ecosystem on top of it (such as ASP.NET Core) pluggable to allow for others. In the end, different consumers will have different goals and being able to plug-in a different library means you can get maximum flexibility in choosing the component that has the best cost/benefit for your app.
Hey @terrajobst. Will the new JSON appear as a netstandard API surface, or just integrated into Core for now?
Hey @terrajobst. Will the new JSON appear as a netstandard API surface, or just integrated into Core for now?
Yes, the question is just which release train it can catch. 2.1 might be too early.
So the JSON parsing bits baked into the framework are planned to be available when v3.0 goes to RTM or will only the integration Apis in ASP.NET Core be complete (with just one implementation - JSON.NET) that will be swappable at a later date?
The plan for 3.0 is as follows:
Stream
based reader/writer, and a serializer.There is an open ended question what the templates for ASP.NET in 3.0 will use. Depending on fidelity we can provide by 3.0 we might have them pull in the Json.NET integration package. However, the goal is to deliver enough fidelity & parity to only depend on the built-in ones by default.
Thanks - that helps clear things up. π
And some additional questions!
If an integration package is used, will it be used throughout the entire ASP.NET Core pipeline or only in some places? I'm assuming Kestrel will always use the internal readers/writers.
Would the Api ergonomics be:
Assuming that this new parser will be used for all the built-in JSON stuff like appSettings.json
, could I put in an early request for comments to be supported?
Thanks.
This is awesome news! Quick question: What packages will this library depend on?
Why to reinvent a wheel that is tested by production customers? If there is a problem with Json.Net, just send a PR as it's open source.
I suppose the problem with Json.NET is that it's not owned by Microsoft, so it has to be replaced. Oh but there is the one already in System.Runtime.Serialization, called DataContractJsonSerializer. Can you use that, or is it just so fun to code new APIs, DIY, that it cannot be avoided?
The reason I'm not very happy with this is that Json.Net supports already edge cases like e.g. F# Discriminated Unions. Not particularly well, but on a level that developers can live with that. Instead any new API's usually forget anything else than the use case of an ASP.NET-website.
@markrendle There is a opt-in setting on JsonReader (work in progress) to allow comments. The configuration system will likely enable that setting by default.
@Thorium Did you actually read the OP? It explains why not JSON.NET, and that JSON.NET will continue to be officially supported with an add-in package.
@JamesNK π
@Thorium Json.NET isn't going away. You aren't losing anything. This is another option for simple and high performance scenarios.
@Thorium Json.NET isn't going away. You aren't losing anything. This is another option for simple and high performance scenarios.
How will the Json generated to be backward compatible?
For example, I'm using SignalR which is using Json.NET in the background. Now, will my F# discriminated unions serialize to similar structures so that I will not be fighting issues with the new Azure Signalr Service (backplane) throwing runtime exceptions because of serializating the structures differently than my server's current SignalR library?
I hope others will pick up the new APIs quickly. Looking at you, @AzureCosmosDB ;-)
Are you planning on including a class like JObject and support for dynamic or this is out of scope for this feature?
I recommend a look in one of this libs:
this could be a really good way to get inspirations.
Will DataContractJsonSerializer
have this new reader/writer used internally ?
I've reached to authors of many of the popular JSON libraries and invited them to review early drafts of this announcement. My hope is that we can work together to create a solid JSON component for the platform while also keeping the ecosystem on top of it (such as ASP.NET Core) pluggable to allow for others.
Is there a reason why the 2nd most popular JSON library after JSON.NET - ServiceStack.Text which has already been refactored to be built on Span
It may be worth reviewing the MIT licensed high-performance https://github.com/neuecc/Utf8Json
This is definitely what we need... my suggestion for the main class name, just use "Json".
@terrajobst I was wondering when this would happen...
I was always wondering why JSON.Net was added as a direct dependency rather than an abstraction (even considering it is the de-facto JSON package for .Net ecosystem).
However, I think add an abstraction for JSON-only is somehow a shoot on your feet. I think a serializer abstraction like we have in Orleans IExternalSerializer in a shape of Microsoft.Extensions.Serialization
or something would be more effective...
Is there any particular reason why make is JSON-only? I see other cases where people can plug other types of serializers...
@galvesribeiro Something like IOutputFormatter
/IInputFormatter
?
@yaakov-h wasn't aware of those... Were are they?
Okey... makes sense now. So where this new JSON-only abstractions comes to play?
The decision to start this undertaking is also a testament about the inefficiency of System.String (UTF-16 String).
I think that the new JSON hooks that will abstract all json handling between asp.net and a json library would look significantly better if you tackle the task to create a utf-8 string BaseType first.
--> Maybe create a System.Utf8String
Yeah... I remember @migueldeicaza saying a while ago that someday, he will make .Net use utf8
strings π
@jges42 @galvesribeiro The proposal to add Utf8String is https://github.com/dotnet/corefx/issues/30503. It seems it's also planned for .Net Core 3.0.
Will these new JSON APIs have dedicated code paths for both Utf8String
and char
/string
, or does the optimization involve flipping the status quo, so that everything that's not UTF-8 will have to be transcoded to it instead? (This doesn't necessarily involve a huge cost since nearly nothing is string
's native UCS-2UTF-16 and still has to be adapted/accounted for, I'm just trying to get an idea of the API surface. Doing this to let Kestrel be more efficient is reasonable; I just hope the design considers more clients than Kestrel.)
@galvesribeiro Actually I think you raise a good point. I think creating an efficient Serialization Framework and an efficient Json Decoder/Encoder are two kind of problems. I know there are some ways to mark a struct as Serializable but I've never seen it used for any Json Serialization.
The Serde project from Rust has actually a good concept by splitting the problem into 2 problems:
A type can either implement Serialize/Deserialize by hand or by a macro(which could be seen as some form of a compiler plugin) which generates the code needed to perform implement those Traits. If a type contains a child type which does not implement those Traits It will even warn at Compile Time. It's a nice concept overall because It means you can just write some data objects an (de)serialize It for any supported format. Writing a format is much easier this way.
I don't think Serde's ways will all work for C# because It doesn't really offer any type specific attributes which could be important for some data structures. So there has to be some work done for this. Also considering AoT Compilation will be very important for some projects(WASM) It should also function well with It.
Here is a link to the Serde docs to make It more clear(Click the bottom 4 Traits to see the concept): https://docs.serde.rs
@mythz The license of ServiceStack.Text is AGPL with some FOSS exception - it would probably prevent people from using it in proprietary software. Also I think it requires clearance from legal for Microsoft employees to even touch it, and any employees who has looked at the source might be barred from working on any other JSON library or related technology.
@poizan42 ServiceStack.Text is dual licensed with both OSS/commercial licenses free to use in both OSS and closed source commercial projects. But the source code licenses are irrelevant as MS is developing their own implementation.
The assertion was that MS was collaborating with the "ecosystem" to develop "pluggable" JSON serializer APIs - if ServiceStack which has been in active development in nearly a decade which is one of the few independent .NET Software suites that has managed to sustain it's own independent healthy community outside of MS within its lifetime, that maintains the 2nd most popular JSON serializer after JSON.NET and what appears to be the 2nd most popular actively developed Web Framework (outside of MS), that runs on more platforms than any MS web framework isn't considered part of the "ecosystem" who is primarily affected by these changes, I'm curious what "ecosystem" are they referring to and why we're being excluded and how many others are being excluded because they're not considered part of the "ecosystem".
I don't understand all this resentment. Asp.net forced you to use a specific version of json.net. They're changing it so you can pick which ever JSON parser you want (or mix it), and there's a default one OOB. ServiceStack should be happy about this change and monitor and provide feedback on this development, rather than just whine about how it had been overlooked, which rarely is an effective way to foster a good community spirit. I personally know many of the .net team members and I'm confident they didn't intend any malice. They are all big proponents of OSS and community work. Personally any GPL derived license would be a big automatic no for me. Apache or MIT for me and my customers or we will move on. No mysterious dual licenses.
Asp.net forced you to use a specific version of json.net
Nope. How so?
I don't understand all this resentment.
Seconded!
I am personally happy that we will finally be able to use the serializer of our choice without having to downloading JSON.Net only to not use it, but still need to ship it because ASP.Net has a hard reference to the library.
(Shameless plug: https://github.com/gregsdennis/Manatee.Json)
@dotMorten
I don't understand all this resentment.
Because you've either not read or understood my comments. Try to responding directly to my comments (i.e. use the quote feature) instead of making up your own narrative.
They're changing it so you can pick which ever JSON parser you want (or mix it)
So like a magic "mix", they'll automatically choose the most optimal API and plugability options that existing .NET serializers are going to be able to plug straight in/out and mix into their internal customizability options with wire-format being exactly the same and everything's just going to work across all serializers? In that case you're right no collaboration or integration testing is needed before APIs are solidified. Or perhaps Serializer implementations are more nuanced with various differences and opinions and everything's not just going to work, not all customization options are going to implemented exactly, the wire-format is not going to be the same and it's not going to be possible to achieve perfect interop between different implementations. The "plugability" that you're glossing over makes a big difference which can determine how much of a rewrite we'll have to do and whether or not it will be possible to support existing and this new implementation.
ServiceStack should be happy about this change and monitor and provide feedback on this development,
Which we've not been given any opportunity to do (or still know how to do), but thanks for letting me know how I should feel. I'd prefer to assess the functionality, interoperability and compatibility of the library before being able to assess the strength of each. Maybe it will be great and it will be easy to support both implementations but from experience interoperating with different serializer implementations is fraught with incompatibilities and corner cases and reliance on different serialization-specific implementations and features. My prediction is that interop between JSON.NET and the default impl will be great since that's their design goal and what's being tested against but other serializers are not going to be as fortunate.
rather than just whine about how it had been overlooked, which rarely is an effective way to foster a good community spirit.
I'm challenging their assertion that they've developed this in collaboration with the "ecosystem", .NET has had a history of killing the existing ecosystem each time they bundle a "default" library, which I'm also expecting will be happening here (I'm struggling to recall a time when bundling a default has ever helped the ecosystem). But regardless, we need to develop seamless integration with whatever they're releasing which I'd like to be able to have access and input to before the API's are frozen. But that's ok I don't expect you to care how it affects the existing frameworks/libraries that need to support existing and future implementations, you're likely only concerned if JSON.NET remains supported or not because that's all that impacts you, but try hold onto your assumptions and letting us know how we should feel about absorbing disruptive changes like this.
i'm struggling to recall a time when bundling a default has ever helped the ecosystem
Oh, come on!
(In the rest I mostly agree with your sentiments)
@mythz I'm surprised that this is causing any problems since we today bundle another 3rd party JSON library into the framework. There are only a handful of places where we bake JSON in and most of them have a provider model that users would reasonably replace (like MVC formatters).
My prediction is that interop between JSON.NET and the default impl will be great since that's their design goal and what's being tested against but other serializers are not going to be as fortunate.
I can already tell you that what we will ship will not support the gamut of features that JSON.NET supports. So that's already not true and in-fact, we expect it to be less capable in some cases (because of performance and scope reasons).
The pluggability mostly already exists today and we have default JSON.NET implementations everywhere. This is just changing that default to be the new JSON serializer instead...
@abatishchev
I really can't recall any, when has embedding or adopting a default implementation in their base framework (or projects) benefited the existing surrounding ecosystem? Everytime I've seen it bundled e.g. NuGet, MVC, ORMs, Unit Testing, Web API, etc it's only ever had a detrimental effect, effectively taking the oxygen and motivation for competing within that space.
There are times like where competing libraries like ASP.NET Ajax have failed to compete where they ended up abandoning it and adopted jQuery, but I don't recall a time where it's ever helped? Note: This is just my observation from closely following .NET after several years, maybe there are examples and I'd be curious to know of some? but from my POV the effects of MS defaults has a detrimental effect on the existing ecosystem of functionality it's replacing.
@mythz benefits for the users from having a default solution by Microsoft is not the same as benefits for the alternative solution's authors. EF is the best ORM in the .NET world and MSTest was better than NUnit back in the day. In my opinion.
But let's don't flood and stick to the subject. Cheers!
@davidfowl This is just changing that default to be the new JSON serializer instead...
I would like to propose that there be no default serializer and require that an implementation be downloaded. If there must be a default, will it be baked into the framework or some separate library (as is the case currently)?
I would like to propose that there be no default serializer and require that an implementation be downloaded. If there must be a default, will it be baked into the framework or some separate library (as is the case currently)?
That's unreasonable as the experience will be subpar. Every modern platform has JSON built in.
@davidfowl Not causing problems now because it's unreleased, but we still need to assess the disruption and scope of work it's going to cause. How much effort will it require to seamlessly support it, will we be able to apply customizations to the new impl to support our existing behavior, will we be able to support the new customization model and APIs, are we able to customize our serializer to support the default configuration/wire-format will the new APIs be able to support both .NET Core and .NET Framework - whilst it's clear ASP.NET Core 3 will abandon .NET Framework, it's not clear if the new APIs are going to use .NET Core only types which will prohibit us from continuing to be able to support both .NET Core and .NET Framework.
I can already tell you that what we will ship will not support the gamut of features that JSON.NET supports. So that's already not true and in-fact, we expect it to be less capable in some cases (because of performance and scope reasons).
I'm only ever expecting it to support a subset of JSON.NET features, e.g. will JSON.NET support the default wire-format? (I'm assuming yes). Will the new impl adopt JSON.NET serialization formats where possible (also assuming yes).
How much effort will it require to seamlessly support it, will we be able to apply customizations to the new impl to support our existing behavior, will we be able to support the new customization model and APIs, are we able to customize our serializer to support the default configuration/wire-format will the new APIs be able to support both .NET Core and .NET Framework.
@mythz I'm not following some of this. I'm trying to figure out how much is this discussing the APIs existing at all vs how they are going to be consumed. Maybe we could look at some concrete scenarios?
@mythz the only real concern somewhat that i see for servicestack would be if this new api is not supported on .net framework classic, that way servicestack won't be able to support both .net core and .net classic, as customer, packages depending on those libraries won't be available in .net framework full . Is that your concern ? i am asking because your concern as a concrete example is not clear.
Also this is a proposal in its very initial stage and the goal it want to achieve looks pretty promising. Constructive criticism is always good for any oss projects.
@mythz benefits for the users from having a default solution by Microsoft is not the same as benefits for the alternative solution's authors.
By ecosystem I'm referring to the surrounding .NET library ecosystem/communities (which presumably is also the "ecosystem" in the OP is referring to) that it replaces which I'd also argue .NET Users also benefit from a healthy ecosystem with a variety of options and more competition (as-is the trait of healthier ecosystems like Python, Java, Node, Ruby, PHP, etc).
EF is the best ORM in the .NET world
Soon after EF was released it quickly took the majority ORM marketshare whilst being over 6x slower than NHibernate whilst arguably supporting less features, excerpt from my 2012 InfoQ interview:
Their latest attempt at an ORM Data Access Layer in Entity Framework has negatively impacted the earlier prominent ORM NHibernate's once thriving community as well. Despite being several times slower than every other Open Source .NET ORM, EF has succeeded in attracting more downloads than all other ORMs combined.
Bear in mind this was pre .NET Core where performance is now a top priority, but it's a historical example of the detrimental effect MS defaults has on existing ecosystems/communities. IMO it's pretty much accepted what happen to existing communities when MS introduces defaults which is why there's been recent push back to revert from shipping defaults that compete with IdentityServer and AutoMapper.
and MSTest was better than NUnit back in the day.
IMO it never was (and R# support of NUnit has always been excellent AFAICR) and the fact we couldn't run it cross-platform on Mono meant libraries that supported cross-platform on Mono (before .NET Core) couldn't use it.
But let's don't flood and stick to the subject. Cheers!
I also don't want to hijack this thread on this either, but needed to state why I disagree with your points.
In relation to this, the primary reason now to use a different serializer than JSON.NET is performance and given the reason for this new default serializer is performance. Since most people just use defaults I'm expecting this to have the most noticable impact on JSON.NET share whilst the primary reason for using an alternative serializer should no longer exist with this faster impl. So basically I also see this having a detrimental effect on the existing (library) ecosystem. IMO a weaker ecosystem of JSON libs is a net negative for .NET (not something most consumers will see they'll just use the defaults and forget about the other options), but that's not my main concern.
@davidfowl @shahid-pk
Despite this I would've actually preferred that this existed 8 years ago as the primary reason for developing ServiceStack.Text was because .NET Framework JSON serializers were extremely slow. But after all this time SS.Text has been extended with a number of features across all our libraries, e.g. customizations for supporting different languages ServiceStack Supports, different JSON customization options in ServiceStack, JSON support in ServiceStack Templates, Complex Type JSON blob support in ServiceStack.Redis etc.
So now I'm focused on assessing on what the impact will be, what the new API and plugability options will look like, can we retrofit existing features onto it, will we be able to adopt as the JSON serializer in SS .NET Core Apps (what will it break), will ServiceStack.Text be able to support the new API, will we be able to support .NET v4.5, will it be able to customize it to support wire-formats of existing deployments, etc. I basically have no idea on the any impact on any of this or what the strategy will be going forward as I haven't had a chance to use or see anything yet. I'll know more answers when I get a chance to use it and I'd obviously like to have the opportunity to test integration and provide feedback and propose changes before the APIs have been frozen.
@mythz
Is there a reason why the 2nd most popular JSON library after JSON.NET - ServiceStack.Text which has already been refactored to be built on Span APIs has been excluded?
The omission wasn't intentional. We've actively searched and worked with JSON library authors as part of the CoreFxLab repo and one of our devs started to benchmark our works against theirs. I believe the list got initially populated by using basic search terms like "json" on NuGet. It looks like your library simply didn't show up. I understand that this can be frustrating or concerning for you but try to understand the situation from our end: our team cannot be expected to know every library under the sun. This announcement is part of our open development model to engage the entire community. The only reason we tend to reach out to smaller groups first is make sure our plans & messaging have a reasonable level of thoughtfulness & quality before we share it with the world. Nothing is final yet. We're actively looking for additional feedback.
I'm curious about which "ecosystem" you're referring to and hoping to "work together" with here?
The .NET ecosystem and in particular the parties interested in JSON processing.
I'd obviously be interested in how compatible these "pluggable" APIs end up being or whether this ends up being another area where the adoption and integration of new MS libraries ends up killing the ecosystem that it's replacing.
The purpose of the planned ASP.NET Core extension points is to enable customers to replace the built-in JSON component with whatever JSON library they want. Of course, ASP.NET has always shipped with "batteries included", i.e. a reasonable default experience. In the past this has been Json.NET and moving forward it's a platform provided component. Given that Json.NET was somewhat hard-wired into ASP.NET the new plan seems net-wise better for people like you; so I'm not entirely sure which part of our plan you think is a threat.
There is an open ended question what the templates for ASP.NET in 3.0 will use.
Isn't it time for the Templates to be modular? Like take vue.js for example.
Creating a new vue app allows you to pick the things you want. Why can't something similar be done for asp.net instead of creating 500 templates to cater for all scenarios.
JSON has become an essential part of virtually all modern .NET applications and in many cases even surpassed the usage of XML. However, .NET hasn't had a (great) built-in way to deal with JSON. Instead we've relied on Json.NET which continues to serve the .NET ecosystem well.
Moving forward, we plan on making some changes to our JSON support:
We need high-performance JSON APIs. We need a new set of JSON APIs that are highly tuned for performance by using
Span<T>
and allows for processing UTF-8 directly without having to transcode to UTF-16string
instances. Both aspects are critical for our web server Kestrel, where throughput is a key requirement.Remove dependency from ASP.NET Core to Json.NET. Today, ASP.NET Core has a dependency on Json.NET. While this provides a tight integration between ASP.NET Core and Json.NET, it also means that application developers cannot freely choose which JSON library they are using. This is also problematic for customers of Json.NET as the version is dictated by the underlying platform. However, Json.NET is frequently updated and application developers often want to -- or even have to -- use a specific version. Thus, we want to remove the dependency from ASP.NET Core 3.0 to Json.NET so that customers can choose which version to use, without fearing they might accidentally break the underlying platform. In addition, this makes it also possible to plug-in an entirely different JSON library.
Provide an ASP.NET Core integration package for Json.NET. Json.NET has basically become the Swiss Army knife of JSON processing in .NET. It provides many options and facilities that allow customers to handle their JSON needs with ease. We don't want to compromise on the Json.NET support customers are getting today, for example, the ability to configure the JSON serialization via the
AddJsonOptions
extension method. Thus, we want to provide the Json.NET integration as a NuGet package that developers can optionally install so they get all the bells and whistles they get from Json.NET today. The other part of this work item is to ensure we have the right extension points so that other parties can provide similar integration packages for their JSON library of choice.Below are more details around this plan.
The need for high-performance JSON APIs
The requirements for the .NET stack have changed a bit since the arrival of .NET Core. Historically, .NET has valued usability and convenience. With .NET Core, we've added a focus on performance, and we've made significant investments to serve high performance needs. And the improvements we made in the popular TechEmpower benchmark are a testament to that.
With .NET Core 2.1, we've added a brand new primitive called Span\ that allows us to represent native memory and arrays in a uniform way. With this type, we've also added a set of parsing and encoding APIs that are much more memory efficient without having to resort to unsafe code.
Part of the work of minimizing allocations is to avoid having to transcode UTF-8 payloads into UTF-16 strings, purely for parsing reasons. Currently, Json.NET is implemented by reading UTF-16. We need the ability to read (and write) JSON documents directly in UTF-8 because most network protocols (including HTTP) use UTF-8.
During .NET Core 2.1 we've learned that updating our existing APIs to leverage
Span<T>
has limits. While we did add a bunch of overloads that accept spans, we also had to produce brand new APIs that are designed around minimizing allocations and dealing with buffers, which we exposed inSystem.Buffers
namespaces. And withSystem.IO.Pipelines
we've also added a programming model that enables developers to share buffers without having to deal with lifetime issues.Based on these experiences we believe in order to support JSON parsing, we'll need to expose a new set of JSON APIs that are specifically geared for high-performance scenarios.
You might wonder why we can't just update Json.NET to include support for parsing JSON using
Span<T>
? Well, James Newton-King -- the author of Json.NET -- has the following to say about that:Move Json.NET integration into a separate NuGet package
Today, you cannot use ASP.NET Core without Json.NET because it is a dependency of ASP.NET Core itself. Over the years, we've received feedback that the dependency can conflict with other libraries that have their own dependency on a different version of Json.NET. In the past, we've considered addressing this issue by using a private copy of Json.NET in ASP.NET. However, this would create problems when developers want to configure Json.NET (for instance, in order to control how the serializer behaves when formatting JSON objects).
Moving forward we'd like to:
Replace the internal usage of Json.NET in ASP.NET Core by the new platform-provided JSON APIs.
Factor the public facing usage of Json.NET into an optional integration package that can be acquired from NuGet.
So the existing integration between ASP.NET Core and Json.NET will continue to be supported, but will be moving out of the platform and into a separate package. However, since the integration is then designed to sit on top of the platform, it will also allow customers to update Json.NET to later versions.
Furthermore, customers who need more performance can also choose to use the new JSON APIs, at the expense of the rich feature set that Json.NET offers.