Please retest with Hemera 2

StarpTech commented 6 years ago

Hi, it would be great if you could update the benchmark with Hemera 2.0. I also think that this benchmark has a bug because I can't reproduce the numbers in a very simple benchmark. For me it looks like that the benchmark utility slow down the application. Thanks!

https://github.com/hemerajs/microservice-benchmark.

icebob commented 6 years ago

Hi, I updated versions & results. I think the benchmark framework is working properly. But it seems Seneca blocked other frameworks, so I moved it to the last position. And I can't update Seneca because 3.4.0 doesn't work with seneca-nats-transport.

StarpTech commented 6 years ago

Hi, I also benchmark the frameworks which support NATS but I can't confirm your results. Do you know what's the discrepance?

StarpTech commented 6 years ago

That's correct seneca-nats-transport does not work with Seneca 3.4.0

icebob commented 6 years ago

From platform info, the only difference is that I'm running it on Win7 :)

StarpTech commented 6 years ago

And you are using Node.Js < 8.4

icebob commented 6 years ago

But I ran the last test on 8.4.0

icebob commented 6 years ago

I will try it on my wife's notebook with Win10

StarpTech commented 6 years ago

Here my benchmark on Node.Js 8.4

uite: Call remote actions
.   Running 'Moleculer'...bench-remote > service.online bench-remote#b91118f6-f74d-4fd0-b490-62eb4276cdc4
√ Moleculer*              838 rps
√ Hemera*               5,493 rps
√ Cote*                18,225 rps
√ Seneca*               3,151 rps

   Moleculer*       -95.4%            (838 rps)   (avg: 1ms)
   Hemera*         -69.86%          (5,493 rps)   (avg: 182μs)
   Cote*                0%         (18,225 rps)   (avg: 54μs)
   Seneca*         -82.71%          (3,151 rps)   (avg: 317μs)
-----------------------------------------------------------------------

Node.Js 6.10.0

Suite: Call remote actions
.   Running 'Moleculer'...bench-remote > service.online bench-remote#15a2fecc-4dce-427b-9d73-f32595684871
√ Moleculer*            2,564 rps
√ Hemera*               4,762 rps
√ Cote*                17,264 rps
√ Seneca*               2,491 rps

   Moleculer*      -85.15%          (2,564 rps)   (avg: 389μs)
   Hemera*         -72.42%          (4,762 rps)   (avg: 209μs)
   Cote*                0%         (17,264 rps)   (avg: 57μs)
   Seneca*         -85.57%          (2,491 rps)   (avg: 401μs)
-----------------------------------------------------------------------

There is a strange bug in moleculer

icebob commented 6 years ago

Thanks, I will investigate it.

danielo515 commented 6 years ago

Here are my test results:

Platform info:
==============
   Darwin 17.5.0 x64
   Node.JS: 10.2.1
   V8: 6.6.346.32-node.8
   Intel(R) Core(TM) i5-3210M CPU @ 2.50GHz × 4

Suite: Call remote actions
✔ Moleculer*              354 rps
✔ Hemera*                 389 rps
✔ Seneca*                 243 rps

   Moleculer*       -9.07%            (354 rps)   (avg: 2ms)
   Hemera*              0%            (389 rps)   (avg: 2ms)
   Seneca*         -37.58%            (243 rps)   (avg: 4ms)
-----------------------------------------------------------------------

I removed the Cote because I feel it is cheationg (does not use NATS) and obscure the actual results comparision, which are the other services using NATS

danielo515 commented 6 years ago

Why are the benchmarks using nats-hemera instead of just hemera ? is nats-hemera something old ?

StarpTech commented 6 years ago

Hi nats-hemera is hemera. The naming shows that hemera can not be used without Nats. Did you consider the latest hemera version (5.6) in your tests?

danielo515 commented 6 years ago

Hello @StarpTech

Did you consider the latest hemera version (5.6) in your tests?

No I didn't. I just installed the dependencies the repo has and tested. But (before your comment, seems we are aligned) I installed the latest version (5.6.0) and re-tested. Here are the results on the same hardware (ran two times):

Suite: Call remote actions
✔ Moleculer*              290 rps
✔ Hemera*                 342 rps
✔ Seneca*                 193 rps

   Moleculer*       -15.4%            (290 rps)   (avg: 3ms)
   Hemera*              0%            (342 rps)   (avg: 2ms)
   Seneca*         -43.52%            (193 rps)   (avg: 5ms)
-----------------------------------------------------------------------

Suite: Call remote actions
✔ Moleculer*              365 rps
✔ Hemera*                 393 rps
✔ Seneca*                 332 rps

   Moleculer*       -7.04%            (365 rps)   (avg: 2ms)
   Hemera*              0%            (393 rps)   (avg: 2ms)
   Seneca*         -15.41%            (332 rps)   (avg: 3ms)
-----------------------------------------------------------------------

Quite similar IMO

StarpTech commented 6 years ago

Yes, LGTM. Thanks.

danielo515 commented 6 years ago

LGTM ?

StarpTech commented 6 years ago

https://www.google.de/amp/s/www.urbandictionary.com/define.php%3fterm=LGTM&amp=true

StarpTech commented 6 years ago

On the second view, I noticed that your rps are very low. This must be due to your old hardware 😄 That's my benchmark:

===========================
  Microservices benchmark
===========================

Platform info:
==============
   Windows_NT 10.0.17134 x64
   Node.JS: 9.8.0
   V8: 6.2.414.46-node.21
   Intel(R) Core(TM) i5-6600K CPU @ 3.50GHz × 4

Suite: Call remote actions
√ Moleculer*            1,529 rps
√ Hemera*               6,044 rps
√ Seneca*               3,037 rps

   Moleculer*       -74.7%          (1,529 rps)   (avg: 653μs)
   Hemera*              0%          (6,044 rps)   (avg: 165μs)
   Seneca*         -49.75%          (3,037 rps)   (avg: 329μs)
-----------------------------------------------------------------------

icebob / microservices-benchmark

Please retest with Hemera 2 #4