Performance Outliers

[GoogleCodeExporter]

What steps will reproduce the problem?

Run fast-serialization benchmarks :-)

What version of the Kryo are you using?
2.2.2

http://fast-serialization.googlecode.com/files/result-1.24.html

deserialization of large arrays and large strings seem broken. Other stuff has 
improved significantly, congrats ;-)

this
http://code.google.com/p/fast-serialization/source/browse/trunk/src/test/java/de
/ruedigermoeller/serialization/testclasses/basicstuff/LargeNativeArrays.java

and this
http://code.google.com/p/fast-serialization/source/browse/trunk/src/test/java/de
/ruedigermoeller/serialization/testclasses/basicstuff/StringPerformance.java

Original issue reported on code.google.com by moru0...@gmail.com on 8 Oct 2013 at 11:09

[GoogleCodeExporter]

Thanks for reporting these issues.

I replied on the mailing list:
https://groups.google.com/forum/#!topic/kryo-users/xvUM5RIIdwo

BTW, I looked at fast-serialization and I'm impressed by its speed. So, I'd 
like to ask a few questions about fast-serialization.

1) Have you tried to run the Kryo unit tests with it? Obviously, it would 
require some adaptations. I'm wondering if fast-serialization can serialize 
just anything what Kryo can serialize or if it has certain limitations. A quick 
look at the code gives an impression that some of the things are pretty 
hard-coded.

2) Does it work on platforms which do not support Unsafe properly, e.g. on 
Android, etc?

3) Does it produce a platform independent serialized representation or is it 
platform-dependent due to the usage of Unsafe (or other reasons) and therefore 
can be deserialized only on the same platform which produced the serialized 
representation?

4) It is currently a GPL code. Is it because you really have something against 
Apache, BSD or MIT licenses? Or have you just picked one of the free licenses, 
which happened to be GPL? 

5) A lot of speed seems to come from the fact that fast-serialization contains 
a lot of code inlined by hand. It seems to help the JIT, but I'm wondring how 
time-consuming is it to inline so many things by hand and to maintain it 
aligned when you change something or discover a bug? In such cases you need to 
fix this bug in all the places where you inlined the code.

Thanks, 
  Leo

Original comment by romixlev on 16 Oct 2013 at 7:03

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

1) From my tests it will serialize more than Kryo since I emulate
completely the JDK seerialization ('compatible mode'), so if there is no
custom serializer I fall back to 'compatible mode'. Kryo cuts a lot of
corners here (no readReplace,writeReplace support etc.). FST should be
completely JDK compatible (and that's the feedback of users). The
"hardoding" impression probably results from FST supporting weird legacy
mechanism's of old JDK versions (e.g. putField, getField).

2) It does not require Unsafe, there is a fallback. (-Dfst.unsafe=false).
However I did not check if it compiles with limited Android API. FST is
targeted at server side messaging I am not willing to sacrifice speed to be
Android compatible.

3) fst always uses X86 byte endianess (with or without unsafe). So one
could use Unsafe on x86 and turn it off on e.g. sparc. (see wiki on this)

4) Its LGPL because I don't want to keep a limited ownership. I don't want
a company to basically hijack fst. I have seen many commercial product
consisting primary of hijacked code. I don't support that, because in
general I like getting paid :-)

5) I am aware of the issues of manual inlining. Its not that bad, just some
selected parts (primitive encoding/decoding). Its manageable and the
tradoff is ok for me.

Original comment by moru0...@gmail.com on 16 Oct 2013 at 9:17

[GoogleCodeExporter]

additon to (1) no i did not try to run kryo unit tests, however its a good
idea as testing is probably the hardest part when implementing a generic
serialization.

2013/10/16 R�diger M�ller <moru0011@gmail.com>

Original comment by moru0...@gmail.com on 16 Oct 2013 at 9:36

[GoogleCodeExporter]

I'm working on implementing object creation without invoking a constructor. 
This is done by setting a corresponding instantiation strategy. 

The problem that I face now is - if I set for a Kryo instance a default 
instantiation strategy that does not invoke a constructor, then our current 
serializers for collections fail. This happens due to the fact that some of the 
collections like ArrayList need to be created using a constructor otherwise 
some of their internal fields are not properly initialized, which leads to NPEs.

Therefore I have a question:
- Should I use an object creation strategy that does not invoke constructors 
only for user-defined classes that use FieldSerializer?
- Or may be I should explicitly set an instantiation strategy that uses no-arg 
constructor for all default serializers that are automatically registered by 
Kryo (this would include all our collection serializers)?

Does anyone have other ideas how to handle this situation?

-Leo

Original comment by romixlev on 17 Oct 2013 at 6:12

[GoogleCodeExporter]

The problem we are trying to solve is that Kryo#newInstantiator tries to use a 
zero argument constructor before using the instantiation strategy, correct? Is 
this a problem that needs solving? People can use 
kryo.getRegistration(type).setInstantiator(...) if they want to customize the 
instantiation of a particular type. Alternatively, Kryo#newInstantiator can be 
overridden.

Must commonly I think people customize creation in a serializer anyway. 
FieldSerializer#create can be overridden, otherwise people can just use "new" 
in Serializer#read.

Original comment by nathan.s...@gmail.com on 19 Oct 2013 at 2:57