Closed RuedigerMoeller closed 10 years ago
dyu
commented
10 years ago Looking at the wiki, json/fastjson/databind is now 2x faster than json/jackson/databind (perf regression on the latter maybe)
@RuedigerMoeller, you might have accidentally excluded smile/protostuff-runtime in the results wiki (it was there before)
dyu
commented
10 years ago Also protostuff-runtime is missing
RuedigerMoeller
commented
10 years ago Yeah, i noticed. I'll check back later this day. Note that some stuff is not running (I am not sure if it was protostuf-runtime, not at home currently). I did not exlcude it, some tests were already commented out, i tried including but they had errors. protobuff is also missing. Note that the previous run was a year ago probably with 1.6.
I also noticed pretty bad effects of hotspot (de-)optimization. E.g. if I run libraries isolated, results differ like +-10%. This is because of increasing polymorphism. So libs running at the beginning of test or are not using subclasses of JDK stuff are in favour, the others are hit by multimorphic calls which cannot be inlined.
dyu
commented
10 years ago "I also noticed pretty bad effects of hotspot (de-)optimization. E.g. if I run libraries isolated, results differ like +-10%."
That's the biggest problem with this benchmark.
Something like http://openjdk.java.net/projects/code-tools/jmh/ could be used to achieve an isolated benchmark ("clean jvm"). I've been using it for my own benchmarks.
NathanSweet
commented
10 years ago There is also Caliper, though I haven't used it. https://code.google.com/p/caliper/
-Nate
On Sat, Mar 8, 2014 at 10:09 AM, David Yu notifications@github.com wrote:
"I also noticed pretty bad effects of hotspot (de-)optimization. E.g. if I run libraries isolated, results differ like +-10%."
That's the biggest problem with this benchmark.
Something like http://openjdk.java.net/projects/code-tools/jmh/ could be used to achieve an isolated benchmark ("clean jvm"). I've been using it for my own benchmarks.
Reply to this email directly or view it on GitHubhttps://github.com/eishay/jvm-serializers/pull/43#issuecomment-37093101 .
RuedigerMoeller
commented
10 years ago Seems like a lot of work ... What do you think if I first take the current test and split it up running each serializer in separate. Then I'll update result aggregation.
Another reasonable thing would be to additionally provide separate XML,JSon and binary and "Full" serializers tests (later on). I would be also neat if serializers would be descriptive regarding their features "getFeatureDescription", so we can generate a comparision matrix directly from running the test. This feature description could include "standard" features as well as free text (e.g. object-to-object copy etc.).
(dyu: nice bass+amp btw ..)
NathanSweet
commented
10 years ago Any improvements you'd like to do are more than welcome! Maybe Eishay or someone can give you commit rights so you don't have to wait on PRs.
-Nate
On Sat, Mar 8, 2014 at 1:17 PM, RuedigerMoeller notifications@github.comwrote:
Seems like a lot of work ... What do you think if I first take the current test and split it up running each serializer in separate. Then I'll update result aggregation.
Another reasonable thing would be to additionally provide separate XML,JSon and binary and "Full" serializers tests (later on). I would be also neat if serializers would be descriptive regarding their features "getFeatureDescription", so we can generate a comparision matrix directly from running the test. This feature description could include "standard" features as well as free text (e.g. object-to-object copy etc.).
(dyu: nice bass+amp btw ..)
Reply to this email directly or view it on GitHubhttps://github.com/eishay/jvm-serializers/pull/43#issuecomment-37096221 .
cowtowncoder
commented
10 years ago I had a look, but it seemed to be more geared towards micro-benchmarks.
I agree, lack of isolation is the big issue.
I have used Japex in the past, as it does isolation. But it is lacking in other areas, and Caliper (for example) has some nice ideas.
-+ Tatu +-
On Sat, Mar 8, 2014 at 11:34 AM, Nathan Sweet notifications@github.comwrote:
There is also Caliper, though I haven't used it. https://code.google.com/p/caliper/
-Nate
On Sat, Mar 8, 2014 at 10:09 AM, David Yu notifications@github.com wrote:
"I also noticed pretty bad effects of hotspot (de-)optimization. E.g. if I run libraries isolated, results differ like +-10%."
That's the biggest problem with this benchmark.
Something like http://openjdk.java.net/projects/code-tools/jmh/ could be used to achieve an isolated benchmark ("clean jvm"). I've been using it for my own benchmarks.
Reply to this email directly or view it on GitHub< https://github.com/eishay/jvm-serializers/pull/43#issuecomment-37093101>
.
Reply to this email directly or view it on GitHubhttps://github.com/eishay/jvm-serializers/pull/43#issuecomment-37095505 .
RuedigerMoeller
commented
10 years ago I have already started to run all tests isolated. Additionally I classified serializers (did cost me several very boring hours :(( ) and added more statis options. E.g. to produce a JSON serializers only chart etc. or only schema/generation free BINARY_CROSSLANGUAGE and so on. Therfore I split running the tests and charting them. Also its much more meaningful to let the test run based on time than on iteration (so the slow serializers do not make you wait half a day), will take some days to finish my work :-)
cowtowncoder
commented
10 years ago Awesome. These sound like good changes, +1 for all. Runtime limited by time is such an obvious thing to do, and slow-running tests were problematic before.
-+ Tatu +-
On Sun, Mar 9, 2014 at 6:11 PM, RuedigerMoeller notifications@github.comwrote:
I have already started to run all tests isolated. Additionally I classified serializers (did cost me several very boring hours :(( ) and added more statis options. E.g. to produce a JSON serializers only chart etc. or only schema/generation free BINARY_CROSSLANGUAGE and so on. Therfore I split running the tests and charting them. Also its much more meaningful to let the test run based on time than on iteration (so the slow serializers do not make you wait half a day), will take some days to finish my work :-)
Reply to this email directly or view it on GitHubhttps://github.com/eishay/jvm-serializers/pull/43#issuecomment-37133694 .
Hi, did the following changes:
updated wiki page (accidentally). Did not realize that github does not fork wiki with project :-) and did not realize i have write access to eishay's wiki ??