MsgPack/Execution Timing Tests

[zachkinstner]

Hey Daniel, I'm considering switching over to RexProClient from RexConnect for my Fabric project. It wouldn't be the easiest transition, but at the moment, I like the idea.

I added some timing investigation in my fork: https://github.com/zachkinstner/rexpro-client/commit/91350ba8f1f50a84a185655516d62eca9c1d4e6a

The results for sending a simple g script:

t: 169.3032ms
t: 23.322ms
t: 32.464ms
t: 30.0475ms
t: 37.8254ms
t: 25.8672ms
t: 32.8269ms
t: 24.7323ms
t: 39.2613ms
t: 27.9117ms

The PackMessage function seems to be the primary bottleneck:

PackMessage: 28.6169ms
PackMessage: 15.4363ms
PackMessage: 15.2981ms
PackMessage: 15.6862ms
PackMessage: 15.6866ms
PackMessage: 15.6064ms
PackMessage: 15.8843ms
PackMessage: 15.9966ms
PackMessage: 16.0411ms
PackMessage: 15.6306ms

[zachkinstner]

Test scenario: Titan (using local Berkeley DB) running on a separate machine, but within my local network.

Do you experience similar results?

I don't have any experience with the MsgPack library. Do you know whether faster libraries exist, or whether there are faster ways to build the message?

[zachkinstner]

Also a consideration -- this was done via test cases in Debug mode.

[zachkinstner]

Notes from some timing tests: https://gist.github.com/zachkinstner/5726670

If we could remove that ~15ms delay for PackMessage, the RexProClient would be even with RexConnect. Still doesn't beat the Rest API.

[dkuppitz]

Hi Zach,

interesting, I will have a deeper look at this at the weekend and see what I can do.

[dkuppitz]

Just had a quick look on your tests. There's no deserialization logic in your RunRestApi test method. You don't even read the response before stopping the execution time.

[dkuppitz]

Same is true for RunRexConn (no deserialization, only a flat JSON string). Only RunRexPro has a fully typed object when the stopwatch stops.

You should also change the logic to measure the times. Something like this:

var sw = Stopwatch.StartNew();
DoStuff();
sw.Stop();
// now the times do not depend on how/where you log

And to be fair, each method should make at least one assertion (after stopping the time), something like this:

dynamic foo = DoStuff();
sw.Stop();
Assert.AreEqual(foo.Bar, "Bar");

[zachkinstner]

Thanks for your suggestions/help with this. I have improved the test cases based on your notes. I also added more granular timing checkpoints, and tried to get these down to "bare-bones" request/response/read steps.

Only RunRexPro has a fully typed object when the stopwatch stops

Using ScriptResponse.Dump() was the only way I found to access the JSON without deserializing into an object. This seemed to be very fast (< 0.1ms). I assume the deserialization was quite fast too, however, since this didn't noticeably affect the times.

The previous RexConnect test was deserializing the JSON into objects, as well (here and in DbResult). I removed this part from the test, this reduced the times by ~2ms.

Do you see any further test/timing issues?

[dkuppitz]

I also wrote some tests and - surprise, surprise - the results were the same. I really didn't expect that. However, I localized the "problem" and there's not much I can do about it. Most time is spend in MsgPacks Pack and Unpack methods.

I've used NewtonSofts (de)serializer for the JSON part and it really seems to be waaaay faster than MsgPack. ServiceStacks implementation could be even faster with less payload when sending JSON requests (I never used ServiceStack, I only know about their benchmark from 2010).

[dkuppitz]

Program + results (really ugly): https://gist.github.com/dkuppitz/5732817

[dkuppitz]

Only the serialization: https://gist.github.com/dkuppitz/5733013

[zachkinstner]

ServiceStacks implementation could be even faster

I use ServiceStack.Text for my Fabric project, I've been very happy with it. I came across these performance tests today:

• http://stackoverflow.com/a/3508631
• http://theburningmonk.com/2011/11/performance-test-json-serializers-part-ii/

You mentioned the upcoming alternate formats for RexPro. I searched around for the thread/issue where that's being discussed, for reference: https://github.com/tinkerpop/rexster/issues/296

[dkuppitz]

I rewrote the whole core, MsgPack is completely removed. I'll push these changes as soon as the new Rexster version is officially published.

I decided to stay with Newtonsofts JSON.NET. First I tried ServiceStack.Text, but it appeared to be so much more complicated then JSON.NET. Anyway, the new implementation is pretty cool.

[zachkinstner]

That sounds great, Daniel. Would you consider pushing to a separate branch in the meantime? I'm not sure if I'll have time to try it out, but it would be nice to have it available.

I'm curious to see what you mean about JSON.NET. I haven't used it before, but I thought the ServiceStack.Text interface was simple enough -- maybe a little verbose:

string JsonSerializer.SerializeToString<T>(T value);
T JsonSerializer.DeserializeFromString<T>(string value);

[dkuppitz]

Here you go: https://github.com/dkuppitz/rexpro-client/tree/rexpro-v1

[zachkinstner]

@dkuppitz, I built a timing utility called RemoteGremlinTests. It's roughly based on the previous tests noted above.

I just posted the first batch of test results. The tests will be difficult/annoying to try to replicate at this point, since I'm currently using some custom builds. That should get better with the release of TinkerPop 2.4.0 and the next Titan.

Initial thoughts from the tests:

• In simple situations, the various options are roughly equivalent.
• In many cases, the RexPro-based options are still not outperforming the Gremlin Extension via HTTP.
• In complex session-based situations, RexConnect can perform upwards of 4x faster than direct RexPro. This is because RexConnect can build up multi-command (session actions, and queries) requests, which avoids the need for a trip to/from the database server for each command.

I'm interested to get some feedback on this, and to see where we can improve and expand these tests.