RuedigerMoeller / fast-serialization

FST: fast java serialization drop in-replacement
Apache License 2.0
1.59k stars 247 forks source link

Making Int2ObjectMap configurable through a factory. #218

Closed angularos closed 7 years ago

angularos commented 7 years ago

This change allows to register a new Int2ObjectMap factory to the configuration while keeping the older implementation as the default. To set up a new factory clients of the library will have to create a factory and set it through the configuration. In addition clearing of the default configuration is now responsible for deciding how to clean itself while maintining implementation details encapsulated.

RuedigerMoeller commented 7 years ago

kewl, let the games begin :), just let me know if you think you came up with a faster solution. As fst has some age, i don't remember exactly which map is the most tied to performance .. afaik its the Identity HM (which has some dirty tweaks). However benchmarks indicate "write" is faster than read which might be related to IdentityHM vs Int2ObjMap ..

OS:Linux
JVM:Oracle Corporation 1.8.0_131
CPU:null os-arch:null
Cores (incl HT):8
1.8.0_131-8u131-b11-2ubuntu1.16.04.3-b11,OpenJDK 64-Bit Server VM,Linux

Test Class: FrequentPrimitives[]

A array[10] of objects with a typical distribution of primitve fields: 2 short Strings, 3 boolean, 6 ints, 2 long, one double.

speed read+write (ns)

size (bytes)


lib                                     read (ns)     write (ns)     total (ns)   size (bytes)
FST 2                                        2767           2000           4767            864
Kryo 4.0.0 Unsafe                            2572           2297           4869            848
Kryo 4.0.0                                   2708           2469           5177            708



Test Class: FrequentPrimitivesExternalizable[]

A class with a typical distribution of primitve fields: 2 short Strings, 3 boolean, 6 ints, 2 long, one double. But implementing Externalizable

speed read+write (ns)

size (bytes)


lib                                     read (ns)     write (ns)     total (ns)   size (bytes)
FST 2                                        2088           1863           3951            882
Kryo 4.0.0 Unsafe                            2731           2606           5337            862
Kryo 4.0.0                                   3031           2918           5949            722



Test Class: FrequentCollections

Measures serialization of most popular collection classes. (HashMap and an ArrayList filled with Integer and Long).

speed read+write (ns)

size (bytes)


lib                                     read (ns)     write (ns)     total (ns)   size (bytes)
FST 2                                       21077          12437          33514           4672
Kryo 4.0.0 Unsafe                           28112          17600          45712           4443
Kryo 4.0.0                                  29280          22994          52274           4619



Test Class: LargeNativeIntArrays

measures performance serializing a large int array, a large long array and a large double array filled with random values. Note that random values destroy any value compression.

speed read+write (ns)

size (bytes)


lib                                     read (ns)     write (ns)     total (ns)   size (bytes)
Kryo 4.0.0 Unsafe                            8078           2211          10289          31210
FST 2                                       25774          17719          43493          31217
Kryo 4.0.0                                  54158          46977         101135          34779



Test Class: LargeAndStrangeStrings

measures serialization of mid size to very large unicode Strings

speed read+write (ns)

size (bytes)


lib                                     read (ns)     write (ns)     total (ns)   size (bytes)
FST 2                                       29005          16063          45068          20002
Kryo 4.0.0                                  76406          56306         132712          17716
Kryo 4.0.0 Unsafe                           83313          52592         135905          17716



Test Class: Primitives

Enums, EnumSets, Date, String, byte, short, int, .. Byte, Character; Short, Integer, ..

speed read+write (ns)

size (bytes)


lib                                     read (ns)     write (ns)     total (ns)   size (bytes)
FST 2                                        1841           1218           3059            302
Kryo 4.0.0 Unsafe                            2086           1524           3610            354
Kryo 4.0.0                                   2126           1564           3690            315



Test Class: Arrays[]

Tests various arrays of int, Dimension, Date, Object. Nested arrays, multidimensional arrays.

speed read+write (ns)

size (bytes)


lib                                     read (ns)     write (ns)     total (ns)   size (bytes)
Kryo 4.0.0 Unsafe                           42509          30165          72674          15644
FST 2                                       45502          32799          78301          15112
Kryo 4.0.0                                  57034          46321         103355           9022



Test Class: CommonCollections

In depth test of collections incl. collections of collections.
(ArrayList, ArrayDeque, ConcurrentLinkedQueue, Vector, TreeSet, LinkedList, TreeMap, ConcurrentHashMap, Hashtable, HashMap). Note: has been modified to workaround Kryo-specific Issue.

speed read+write (ns)

size (bytes)


lib                                     read (ns)     write (ns)     total (ns)   size (bytes)
FST 2                                       43879          33244          77123           8345
Kryo 4.0.0 Unsafe                           91395          42884         134279           9408
Kryo 4.0.0                                  91496          46189         137685           8323



Test Class: Trader

Measures serialization of a typical 'Enterprise-Object' using Value-Classes instead of primitive tpyes.

speed read+write (ns)

size (bytes)


lib                                     read (ns)     write (ns)     total (ns)   size (bytes)
FST 2                                       57092          52433         109525           6072
Kryo 4.0.0 Unsafe                           63975          55706         119681           5928
Kryo 4.0.0                                  66084          63404         129488           5919



Test Class: ManyClasses

Tests speed serializing a complex object graph of many different classes with few primitive fields.

speed read+write (ns)

size (bytes)


lib                                     read (ns)     write (ns)     total (ns)   size (bytes)
FST 2                                        5299           4099           9398           1061
Kryo 4.0.0 Unsafe                            6907           4544          11451           1283
Kryo 4.0.0                                   6931           4774          11705           1220



Test Class: ExternalizableTest

Performance of Externalizable objects.

speed read+write (ns)

size (bytes)


lib                                     read (ns)     write (ns)     total (ns)   size (bytes)
FST 2                                       54130          48775         102905           8971
Kryo 4.0.0 Unsafe                           67002          62765         129767           7507
Kryo 4.0.0                                  67790          67016         134806           4733



Test Class: BigObject

A bigger object graph consisting of some of the test objects at once

speed read+write (ns)

size (bytes)


lib                                     read (ns)     write (ns)     total (ns)   size (bytes)
FST 2                                     3218901        2681658        5900559         401976
Kryo 4.0.0                                3794706        3304696        7099402         321022
Kryo 4.0.0 Unsafe                         3630378        3477519        7107897         394636



Test Class: HeavyNesting

Heavily nested Objects

speed read+write (ns)

size (bytes)


lib                                     read (ns)     write (ns)     total (ns)   size (bytes)
FST 2                                       59684          57574         117258           4748
Kryo 4.0.0                                  63801          75881         139682           5009
Kryo 4.0.0 Unsafe                           87843          83266         171109           7076



Test Class: MediaContent

measures serializing the cyclefree object structure used by one of the most known encoding/decoding tests

speed read+write (ns)

size (bytes)


lib                                     read (ns)     write (ns)     total (ns)   size (bytes)
FST 2                                        1615           1490           3105            322
Kryo 4.0.0 Unsafe                            1713           1465           3178            309
Kryo 4.0.0                                   1704           1508           3212            288



Test Class: SmallThing

only one int. benches pure init time

speed read+write (ns)

size (bytes)


lib                                     read (ns)     write (ns)     total (ns)   size (bytes)
Kryo 4.0.0                                    111            113            224              2
Kryo 4.0.0 Unsafe                             123            114            237              5
FST 2                                         136            143            279              3



Test Class: LargeFPArrays

measures performance serializing a large flaot and a large double array filled with random values.

speed read+write (ns)

size (bytes)


lib                                     read (ns)     write (ns)     total (ns)   size (bytes)
FST 2                                       25167          13736          38903          20812
Kryo 4.0.0 Unsafe                           97348           1449          98797          20807
Kryo 4.0.0                                 110780          18443         129223          20807



Test Class: BasicJSon

speed read+write (ns)

size (bytes)


lib                                     read (ns)     write (ns)     total (ns)   size (bytes)
Kryo 4.0.0 Unsafe                            2331           1284           3615            222
Kryo 4.0.0                                   2380           1879           4259            225
FST 2                                        2837           2280           5117            420



Test Class: JSonCollection

a map of String=>int and a list holding a pojo, some numbers and a string

speed read+write (ns)

size (bytes)


lib                                     read (ns)     write (ns)     total (ns)   size (bytes)
FST 2                                        1379            917           2296            200
Kryo 4.0.0 Unsafe                            1918           1439           3357            276
Kryo 4.0.0                                   2071           1501           3572            234



Test Class: ShortRemoteCall

measures overhead of stream initialization+classname decoding. Only one very short object is serialized

speed read+write (ns)

size (bytes)


lib                                     read (ns)     write (ns)     total (ns)   size (bytes)
FST 2                                         283            241            524             25
Kryo 4.0.0 Unsafe                             469            380            849             48
Kryo 4.0.0                                    525            443            968             41


carterkozak commented 6 years ago

Hi @RuedigerMoeller, any chance we can get a tagged release with this change (and the default FST int object map resize algo!)? Not a rush, but it would certainly be appreciated when you have free time.

Thanks again!

RuedigerMoeller commented 6 years ago

y, will release tomorrow