Open zachkinstner opened 11 years ago
Test scenario: Titan (using local Berkeley DB) running on a separate machine, but within my local network.
Do you experience similar results?
I don't have any experience with the MsgPack library. Do you know whether faster libraries exist, or whether there are faster ways to build the message?
Also a consideration -- this was done via test cases in Debug
mode.
Notes from some timing tests: https://gist.github.com/zachkinstner/5726670
If we could remove that ~15ms delay for PackMessage
, the RexProClient would be even with RexConnect. Still doesn't beat the Rest API.
Hi Zach,
interesting, I will have a deeper look at this at the weekend and see what I can do.
Just had a quick look on your tests. There's no deserialization logic in your RunRestApi test method. You don't even read the response before stopping the execution time.
Same is true for RunRexConn (no deserialization, only a flat JSON string). Only RunRexPro has a fully typed object when the stopwatch stops.
You should also change the logic to measure the times. Something like this:
var sw = Stopwatch.StartNew();
DoStuff();
sw.Stop();
// now the times do not depend on how/where you log
And to be fair, each method should make at least one assertion (after stopping the time), something like this:
dynamic foo = DoStuff();
sw.Stop();
Assert.AreEqual(foo.Bar, "Bar");
Thanks for your suggestions/help with this. I have improved the test cases based on your notes. I also added more granular timing checkpoints, and tried to get these down to "bare-bones" request/response/read steps.
Only RunRexPro has a fully typed object when the stopwatch stops
Using ScriptResponse.Dump()
was the only way I found to access the JSON without deserializing into an object. This seemed to be very fast (< 0.1ms). I assume the deserialization was quite fast too, however, since this didn't noticeably affect the times.
The previous RexConnect test was deserializing the JSON into objects, as well (here and in DbResult). I removed this part from the test, this reduced the times by ~2ms.
Do you see any further test/timing issues?
I also wrote some tests and - surprise, surprise - the results were the same. I really didn't expect that. However, I localized the "problem" and there's not much I can do about it. Most time is spend in MsgPacks Pack and Unpack methods.
I've used NewtonSofts (de)serializer for the JSON part and it really seems to be waaaay faster than MsgPack. ServiceStacks implementation could be even faster with less payload when sending JSON requests (I never used ServiceStack, I only know about their benchmark from 2010).
Program + results (really ugly): https://gist.github.com/dkuppitz/5732817
Only the serialization: https://gist.github.com/dkuppitz/5733013
ServiceStacks implementation could be even faster
I use ServiceStack.Text for my Fabric project, I've been very happy with it. I came across these performance tests today:
You mentioned the upcoming alternate formats for RexPro. I searched around for the thread/issue where that's being discussed, for reference: https://github.com/tinkerpop/rexster/issues/296
I rewrote the whole core, MsgPack is completely removed. I'll push these changes as soon as the new Rexster version is officially published.
I decided to stay with Newtonsofts JSON.NET. First I tried ServiceStack.Text, but it appeared to be so much more complicated then JSON.NET. Anyway, the new implementation is pretty cool.
That sounds great, Daniel. Would you consider pushing to a separate branch in the meantime? I'm not sure if I'll have time to try it out, but it would be nice to have it available.
I'm curious to see what you mean about JSON.NET. I haven't used it before, but I thought the ServiceStack.Text interface was simple enough -- maybe a little verbose:
string JsonSerializer.SerializeToString<T>(T value);
T JsonSerializer.DeserializeFromString<T>(string value);
@dkuppitz, I built a timing utility called RemoteGremlinTests. It's roughly based on the previous tests noted above.
I just posted the first batch of test results. The tests will be difficult/annoying to try to replicate at this point, since I'm currently using some custom builds. That should get better with the release of TinkerPop 2.4.0 and the next Titan.
Initial thoughts from the tests:
I'm interested to get some feedback on this, and to see where we can improve and expand these tests.
Hey Daniel, I'm considering switching over to RexProClient from RexConnect for my Fabric project. It wouldn't be the easiest transition, but at the moment, I like the idea.
I added some timing investigation in my fork: https://github.com/zachkinstner/rexpro-client/commit/91350ba8f1f50a84a185655516d62eca9c1d4e6a
The results for sending a simple
g
script:The
PackMessage
function seems to be the primary bottleneck: