Benchmark NetJSON with the JSON serializer comes with .NET 5?

[wmjordan]

.NET 5 has been released for quite a few months and it has a JSON serializer which sounds to be quite good at performance. Is it possible to add that to the benchmark and see how it performs?

[rpgmaker]

Sure, that is a good idea. I wonder if there is an existing benchmark online that already does this comparison for me.

[rpgmaker]

I think the JSON serializer in .net 5 is the same one mentioned here from 2019 - https://michaelscodingspot.com/the-battle-of-c-to-json-serializers-in-net-core-3/ if so then netjson should be faster than it since jil was faster than it.

[wmjordan]

The .NET 5 one has been optimized further more. Please take a look at this blog.

[wmjordan]

In the above blog there is a link to the benchmark: https://github.com/dotnet/performance/tree/main/src/benchmarks/micro/libraries/System.Text.Json/Serializer

[rpgmaker]

I created a simple code to test it

• Calling NetJSON then Microsoft JSON

• First Run Completed NetJSON in 4080 milliseconds Completed Microsoft Json in 7022 milliseconds

• Second Run Completed NetJSON in 3775 milliseconds Completed Microsoft Json in 5440 milliseconds

• Calling Microsoft JSON then NetJSON

• First Run Completed Microsoft Json in 4929 milliseconds Completed NetJSON in 3137 milliseconds

• Second Run Completed Microsoft Json in 6511 milliseconds Completed NetJSON in 2637 milliseconds

Code Snippet

public class SimpleObject
    {
        public string Name { get; set; }
        public int ID { get; set; }
    }
    class Program
    {
        static int count = 10000000;
        static string netjson = null;
        static string json = null;
        static void Main(string[] args)
        {
            var obj = new SimpleObject { ID = 10, Name = "Performance" };

            Test("NetJSON", () => netjson = NetJSON.NetJSON.Serialize(obj));
            Test("Microsoft Json", () => json = JsonSerializer.Serialize(obj));
        }

        public static void Test(string name, Action action)
        {
            Stopwatch watch = new Stopwatch();

            GC.Collect();
            //warm up
            action();

            watch.Start();

            for (int i = 0; i < count; i++)
            {
                action();
            }

            watch.Stop();
            Console.WriteLine($"Completed {name} in {watch.ElapsedMilliseconds} milliseconds");
        }
    }

[rpgmaker]

Seem netjson is still faster

[rpgmaker]

I ran another run in release mode and got the following

Completed NetJSON in 2342 milliseconds Completed Microsoft Json in 6212 milliseconds

[rpgmaker]

So according to this, it seem netjson is still about 3x faster than it :)

[wmjordan]

That's cool!

I suggest deploying BenchmarkDotNet to benchmark them. It is quite easy to setup new test cases and we can say good bye to Stopwatch

[rpgmaker]

Will take a look at it and see what the result looks like tomorrow

[rpgmaker]

This is the result after running it

// BenchmarkRunner: Finish

// Export BenchmarkDotNet.Artifacts\results\ConsoleApp1.SimpleObjectBenchmark-report.csv BenchmarkDotNet.Artifacts\results\ConsoleApp1.SimpleObjectBenchmark-report-github.md BenchmarkDotNet.Artifacts\results\ConsoleApp1.SimpleObjectBenchmark-report.html

// Detailed results SimpleObjectBenchmark.NetJson: DefaultJob Runtime = .NET Core 5.0.0 (CoreCLR 5.0.20.51904, CoreFX 5.0.20.51904), X64 RyuJIT; GC = Concurrent Workstation Mean = 219.910 ns, StdErr = 4.005 ns (1.82%), N = 96, StdDev = 39.246 ns Min = 160.346 ns, Q1 = 193.979 ns, Median = 215.568 ns, Q3 = 241.889 ns, Max = 327.742 ns IQR = 47.909 ns, LowerFence = 122.115 ns, UpperFence = 313.753 ns ConfidenceInterval = [206.307 ns; 233.512 ns] (CI 99.9%), Margin = 13.602 ns (6.19% of Mean) Skewness = 0.67, Kurtosis = 3.15, MValue = 3.31 -------------------- Histogram -------------------- [159.416 ns ; 184.675 ns) | @@@@@@@@@@@@@@@@@@@ [184.675 ns ; 206.918 ns) | @@@@@@@@@@@@@@@@@@@ [206.918 ns ; 229.417 ns) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@ [229.417 ns ; 240.997 ns) | @@@@ [240.997 ns ; 263.496 ns) | @@@@@@@@@@@@@@@ [263.496 ns ; 290.445 ns) | @@@@@@ [290.445 ns ; 311.462 ns) | @ [311.462 ns ; 333.961 ns) | @@@@

SimpleObjectBenchmark.MicrosoftJson: DefaultJob Runtime = .NET Core 5.0.0 (CoreCLR 5.0.20.51904, CoreFX 5.0.20.51904), X64 RyuJIT; GC = Concurrent Workstation Mean = 481.078 ns, StdErr = 5.410 ns (1.12%), N = 92, StdDev = 51.890 ns Min = 410.196 ns, Q1 = 444.771 ns, Median = 463.468 ns, Q3 = 502.370 ns, Max = 624.837 ns IQR = 57.598 ns, LowerFence = 358.374 ns, UpperFence = 588.767 ns ConfidenceInterval = [462.681 ns; 499.475 ns] (CI 99.9%), Margin = 18.397 ns (3.82% of Mean) Skewness = 1.17, Kurtosis = 3.42, MValue = 2.26 -------------------- Histogram -------------------- [407.405 ns ; 436.178 ns) | @@@@@@@@@@ [436.178 ns ; 466.350 ns) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ [466.350 ns ; 480.505 ns) | @@@@ [480.505 ns ; 510.678 ns) | @@@@@@@@@@@@@@@@@@ [510.678 ns ; 545.092 ns) | @@@@@ [545.092 ns ; 575.264 ns) | @@@@@@@ [575.264 ns ; 613.760 ns) | @@@@@@ [613.760 ns ; 639.923 ns) | @

// Summary

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19042 Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R), 1 CPU, 8 logical and 4 physical cores .NET Core SDK=5.0.100 [Host] : .NET Core 5.0.0 (CoreCLR 5.0.20.51904, CoreFX 5.0.20.51904), X64 RyuJIT DefaultJob : .NET Core 5.0.0 (CoreCLR 5.0.20.51904, CoreFX 5.0.20.51904), X64 RyuJIT

Method	Mean	Error	StdDev	Median
NetJson	219.9 ns	13.60 ns	39.25 ns	215.6 ns
MicrosoftJson	481.1 ns	18.40 ns	51.89 ns	463.5 ns

[rpgmaker]

Code for the benchmark

    public class SimpleObject
    {
        public string Name { get; set; }
        public int ID { get; set; }
    }

    public class SimpleObjectBenchmark
    {
        private readonly SimpleObject obj;

        public SimpleObjectBenchmark()
        {
            obj = new SimpleObject { ID = 10, Name = "Performance" };
        }

        [Benchmark]
        public string NetJson() => NetJSON.NetJSON.Serialize(obj);

        [Benchmark]
        public string MicrosoftJson() => JsonSerializer.Serialize(obj);
    }

    class Program
    {
        static void Main(string[] args)
        {
            var summary = BenchmarkRunner.Run<SimpleObjectBenchmark>();
        }
    }

[rpgmaker]

The result is the same. Almost 3x faster still.

[wmjordan]

Possible to upload your benchmark project to the repository?

Let's add more stuff to benchmark :)

[rpgmaker]

I will try to add it tonight.

[rpgmaker]

Utf8 seem to be have better result. I guess it is time to do some more optimization or maybe not. Since Utf8Json uses byte array and does not deal with string directly. So I might add a byte array utf8 version of serialize and deserialize and compare again.

// Summary

BenchmarkDotNet=v0.13.0, OS=Windows 10.0.19042.985 (20H2/October2020Update) Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R), 1 CPU, 8 logical and 4 physical cores .NET SDK=5.0.100 [Host] : .NET 5.0.0 (5.0.20.51904), X64 RyuJIT DefaultJob : .NET 5.0.0 (5.0.20.51904), X64 RyuJIT

Method	Mean	Error	StdDev	Median
NetJson	219.8 ns	16.36 ns	45.35 ns	204.8 ns
MicrosoftJson	503.0 ns	23.84 ns	66.46 ns	490.9 ns
FastestUtf8Json	150.8 ns	5.70 ns	15.61 ns	147.3 ns

[rpgmaker]

The code is in, so we can make improvement i suppose.

[wmjordan]

Good. The fun begins now :)

[wmjordan]

I am refactoring the benchmark project. I introduced a generic type so we don't have to repeat [Benchmark] methods.

public abstract class Benchmark<TObj>
{
    private readonly TObj obj;

    protected Benchmark(TObj instance)
    {
        obj = instance;
    }

    [Benchmark]
    public string NetJson() => NetJSON.Serialize(obj);

    [Benchmark]
    public string MicrosoftJson() => JsonSerializer.Serialize(obj);

    [Benchmark]
    public byte[] FastestUtf8Json() => Utf8Json.JsonSerializer.Serialize(obj);
}

At first glance, it seems that it is "unfair" to let Utf8Json.JsonSerializer returning a byte array yet the other two returns string. However, after a second thought, it really has its advantage. I have a Web API server which returns JSON responses, byte arrays could be send to the client directly via TCP sockets. I don't have to encode the strings to byte arrays on the server. This could save quite sometime and make the overall performance even better!

[rpgmaker]

Though it is unfair it is still nice to have. It is like having the mix of json and binary serialization.

I guess I could use string builder and just extract only the buffer . Not sure what the cost of converting to string from char array is in the overall benchmark.

Could you convert the changes you made into a pull request? Thanks

[wmjordan]

Yes, I am about to make the PR.

And I found that in some situation NetJSON was 3 times SLOWer than the other two. Please check out the StringDictionaryBenchmark.

And here's the result on my laptop.

BenchmarkDotNet=v0.13.0, OS=Windows 10.0.19042.964 (20H2/October2020Update) Intel Core i5-1035G4 CPU 1.10GHz, 1 CPU, 8 logical and 4 physical cores .NET SDK=5.0.203 [Host] : .NET 5.0.6 (5.0.621.22011), X64 RyuJIT DefaultJob : .NET 5.0.6 (5.0.621.22011), X64 RyuJIT

Method	Mean	Error	StdDev
NetJson	6.387 us	0.1380 us	0.4025 us
MicrosoftJson	1.494 us	0.0298 us	0.0752 us
FastestUtf8Json	1.187 us	0.0234 us	0.0509 us

[rpgmaker]

That is very interesting. I wonder why.

[wmjordan]

To investigate it, I changed the StringDictionaryBenchmark to serialize Dictionary<string,string>, afterwards things were quite different.

Method	Mean	Error	StdDev
NetJson	354.2 ns	6.90 ns	9.45 ns
MicrosoftJson	557.5 ns	10.44 ns	9.26 ns
FastestUtf8Json	394.1 ns	6.60 ns	6.17 ns

The problem may lie in serializing boxed value type instances. To verify the assumption, two benchmarks were performed.

A benchmark which serialized an boxed integer, where NetJSON was the worst.

public class BoxedInt32Benchmark : Benchmark<object>
{
    public BoxedInt32Benchmark() : base(3) {
    }
}

Method	Mean	Error	StdDev
NetJson	937.28 ns	18.576 ns	29.997 ns
MicrosoftJson	160.91 ns	3.253 ns	4.343 ns
FastestUtf8Json	61.28 ns	1.266 ns	1.458 ns

In another benchmark where the integer was not boxed, NetJSON performed much better.

public class Int32Benchmark : Benchmark<int>
{
    public Int32Benchmark() : base(3) {
    }
}

Method	Mean	Error	StdDev
NetJson	27.83 ns	0.627 ns	0.919 ns
MicrosoftJson	163.11 ns	3.293 ns	8.381 ns
FastestUtf8Json	26.61 ns	0.570 ns	1.276 ns

[rpgmaker]

That make sense. I can focus on improvement of the unboxing and boxing in that case

[rpgmaker]

Closing this issue since we now have a benchmark project @wmjordan