Alois-xx
commented
6 years ago -
Regarding Benchmark.NET I have mixed feelings about it. It is nice for micro benchmarks but the serializer tests are not really micro benchmarks. I explictely want to test O(n) behavior. For that I need many invocations of a method with different parameters. I prefer a real profiler which has nearly zero overhead like ETW where I get GC, excpetions, VirtualAlloc and such all for free and I can later drill into the data to compare the serializers not only by the timing but also why a specific method was slow.
-
I have added with commit d260a7f also some reference tracking tests which contain interesting results. see https://aloiskraus.wordpress.com/2018/05/06/serialization-performance-update-with-net-4-7-2/ The ReferenceBookShelf contains a dictionary of DateTime objects which share the same string reference. That should bring nearly all types you wanted to see as well (at least floats). This test did show that FastJSON cannot round trip DateTime objects and JIL cannot cope with Dictionaries having DateTime objects as keys.
-
The article table formatting is fixed now. The crazy wordpress template is always squishing my tables.
-
The updated article should address the other missing information now as well.
Is the current state sufficient for you now? The goal is that you can use the project and try some new serializers with your custom objects out to see if the performance is so good or not. Bringing many more tests for different data types is in my opinion of no use since you will quickly run into graph and information overload. It is better to try your specific objects with the test suite and check for yourself.
It will be much better if BenchmarkDotNet will be utilized to perform the measurements and produce the comparison charts. Currently, I have a too high deviation of the test run results which, from my experience, could be eliminated by using BenchmarkDotNet (it helps to produce the consistent results with the various statistical information - and also it has many more features which might help with tests bootstrapping (runtime selection, including Mono), preparation (warmup), operation (such as measuring separately the first run and the repeated performance (without counting the first run) without any extra work/code copypasting), processing the measurement results and rendering them in a chart automatically). I know this is a big thing to ask (considering how everything is already designed and implemented), but it can make this test suite perfect!
The current test suite is very light and too specific - just two classes one of which contains a single list of the other class instances, 95% of data volume are strings. You're mostly testing the throughput (which is also an important metric), but a real world scenario might include much more data kinds. Also, I don't feel it's fair to compare tree serializers with full object graph serializers (including .NET BinaryFormatter which lead you to write the test suite and article!) - the use cases are very different. I would suggest making two (or maybe even three, as not all full object graph serializers are full-featured) separate groups of tests for these two different kinds of serializers (though, object graph serializers can continue participating in tests for tree serializers). The current test case need to be updated - same test as now (bookshelf-book), but more different kinds of data. Consider adding to Book class the following fields:
List<BookTag>, and BookTag could be a enum with few values just for diversity sake)ulong) might be a good idea, I don't feel usingGUIDis good idea as it's itself very slow). And let's make it a readonly field - because that's pretty often the case!double, could be in range 0-1 where 1 means "new" and 0 means "unreadable", just for example)DateTime)TimeSpan). And as with the unique book code, let's make this a readonly property.uint IDandstring Name, total size of owners collection could be just 10 owners. I believe it will make the test much less biased towards the tree binary serializers as object graph serializers can reuse Owner instances where tree serializers will simply write the same objects again and again, wasting performance and unnecessarily increasing the output size. Arguably, it's a real world scenario.I propose to make a second test specifically for object graph serializers - with much more references (though not sure about the circular references - it might break compatibility with too many binary serializers from the list) and full verifications that the objects references are indeed restored (so we can be sure that a serializer is properly configured and doesn't produce clones where reference tracking is required). I don't have any good ideas yet, but we can invent something based on the bookshelf-book-owner case - say, add cross-references between owner and books she ever read (but it will be hard to avoid ciclic references in that case). Another important thing is full C# support, including generics and null handling. I would prefer to see a third test case (or drop the proposed second test case in favor of this!) - which will be used only for serializers which can 99.9% replace
BinaryFormatter- including generics support, various collection types, support for fields ofobjecttype (and boxed values), proper null handling, tuples (and ValueTuples), inheritance/interfaces support (polymorpic serialization, and ensure that private members of parent classes are properly (de)serialized), cyclic references, structs with readonly fields and other pretty common stuff for every experienced .NET developer. Though, no delegates and comparers (such as stored in Dictionary and HashSet) - AFAIK, there are no alternatives toBinaryFormatterwhich can handle that. For everything else, at least Wire/Hyperion, AqlaSerializer and my own serializer will satisfy all these requirements and it will be very interesting to compare them against each other andBinaryFormatter.The article formatting is not good in a few places, especially the table of serializers feature comparison and examples of serializers output. Maybe because I'm using Firefox, but reading "Default Serializer Type" column is impossible to me - it's 1 char width!
Again, about the serializers comparison table: consider adding "supports C# generics" column and "custom requirements" (for example, "requires decoration of types and fields with its own custom attributes", "all fields/properties must be non-readonly") columns. For people who are looking for a replacement to
BinarySerializerit will be very helpful!Regards!
UPD. I understand that the initial intention was partly to demonstrate how slowly
BinaryFormatterperformed when objects count exceeded the defined threshold. But this test case will be soon irrelevant (thanks to your report!) except the legacy cases. You can keep the current test case, but I think developers will be more interested in a more general test case to compare serializers.Another benefit of BenchmarkDotNet will be memory allocation measurements and GC count. I can definitely say that it's possible to write a C# binary serializer which doesn't perform (almost) any allocations during (de)serialization (other than allocation of the instantiated objects themselves during the deserialization) as this is what I've already done. Some allocations are required for any object graph serializer (to track references, to buffer chars, etc) but these allocations might be done in a session object which could be reused for future calls of the (de)serialize method and with enough capacity re-allocations are not required.