Phoenix and Plug benchmark improvs

[kelvinst]

fix for phoenix

[kelvinst]

I'm also adding some performance improvements for plug and phoenix on this PR soon.

[kelvinst]

Done! Now Plug and Phoenix are in prod env. ;)

[OvermindDL1]

Yeah, running Plug in prod instead of dev is definitely smarter. I did tests with a couple with Plug in prod instead of dev and got this:

Language (Runtime)        Framework (Middleware)          Max [sec]       Min [sec]       Ave [sec]
------------------------- ------------------------- --------------- --------------- ---------------
rust                      iron                             4.559537        4.498832        4.534531
elixir                    plug                             5.444713        5.185388        5.326322
crystal                   router_cr                        9.139290        8.827565        9.007032
node                      express                         32.212435       31.668852       31.824345

I'm not sure why router_cr on the chart is showing to be so much faster than rust, I could not replicate that at all.

Also the Elixir/Plug one might be a touch bit faster if it is warmed up first, but that might already be done so if so ignore this sentence. :-)

EDIT: For note, my tests above were done on a very very old AMD Phenom ][ at 3.6ghz 6 native cores with 16 gigs ram freshly rebooted with Ubuntu 17.04 and entirely unused (not even X GUI running)

[OvermindDL1]

Added Phoenix latest pull above:

Language (Runtime)        Framework (Middleware)          Max [sec]       Min [sec]       Ave [sec]
------------------------- ------------------------- --------------- --------------- ---------------
rust                      iron                             4.559537        4.498832        4.534531
elixir                    plug                             5.444713        5.185388        5.326322
elixir                    phoenix                          5.792088        5.673842        5.752746
crystal                   router_cr                        9.139290        8.827565        9.007032
node                      express                         32.212435       31.668852       31.824345

[OvermindDL1]

Don't forget to update the README.md as well as it states to do for PR's.

[kelvinst]

@OvermindDL1, the README is already with Phoenix, it was added by another PR, this one is just a performance improvement.

[OvermindDL1]

I ran some more, still sorted from fastest to slowest as top to bottom:

Language (Runtime)        Framework (Middleware)          Max [sec]       Min [sec]       Ave [sec]
------------------------- ------------------------- --------------- --------------- ---------------
rust                      iron                             4.559537        4.498832        4.534531
rust                      nickel                           5.253805        5.165856        5.207764
elixir                    plug                             5.444713        5.185388        5.326322
elixir                    phoenix                          5.792088        5.673842        5.752746
rust                      rocket                           5.977488        5.849199        5.914374
crystal                   router_cr                        9.139290        8.827565        9.007032
crystal                   kemal                           10.316980        9.544119       10.050472
node                      express                         32.212435       31.668852       31.824345
ruby                      roda                            66.817984       63.877961       65.617340
ruby                      sinatra                        147.016259      144.732534      146.124146
ruby                      rails                          477.844824      475.863197      476.804092

I could not make go because:

╰─➤  make go
go get -u github.com/labstack/echo
package context: unrecognized import path "context" (import path does not begin with hostname)
Makefile:60: recipe for target 'echo' failed
make: *** [echo] Error 1

I could not make swift because:

╰─➤  make swift                                                                                                     1 ↵ cd swift/vapor; swift build --configuration release
error: unsatisfiable
Makefile:103: recipe for target 'vapor' failed
make: *** [vapor] Error 1

I'm getting some very different results from the chart on the repo, I'm wondering if I have a higher amount of (though slower) cores thus the concurrent ones work better?

[OvermindDL1]

And for note:

╰─➤  go version
go version go1.6.2 linux/amd64

╰─➤  swift --version
Swift version 3.1.1 (swift-3.1.1-RELEASE)
Target: x86_64-unknown-linux-gnu

[OvermindDL1]

I made a PHP file of (this is to follow the requirements in the README.md precisely):

<?php
$uri = $_SERVER['REQUEST_URI'];
if($uri == "/") die("");
else if($uri == "/user") die("");
else if(substr($uri, 0, 6) == "/user/") die(substr($uri, 6));
?>

And tested with that using the hhvm php server since that is the only one I have installed at the moment, added the results to the rest here:

Language (Runtime)        Framework (Middleware)          Max [sec]       Min [sec]       Ave [sec]
------------------------- ------------------------- --------------- --------------- ---------------
rust                      iron                             4.559537        4.498832        4.534531
rust                      nickel                           5.253805        5.165856        5.207764
elixir                    plug                             5.444713        5.185388        5.326322
elixir                    phoenix                          5.792088        5.673842        5.752746
rust                      rocket                           5.977488        5.849199        5.914374
crystal                   router_cr                        9.139290        8.827565        9.007032
crystal                   kemal                           10.316980        9.544119       10.050472
php                       hhvm                            18.855467       18.748603       18.801522
node                      express                         32.212435       31.668852       31.824345
ruby                      roda                            66.817984       63.877961       65.617340
ruby                      sinatra                        147.016259      144.732534      146.124146
ruby                      rails                          477.844824      475.863197      476.804092

I'm impressed, PHP is pretty dang fast, especially considering it has to hit the filesystem to check if the file's been changed on every request. PHP utterly blows Ruby away in speed, but still does not beat crystal/elixir/rust...

[tbrand]

@kelvinst That's really awesome!! I'll try this.

@OvermindDL1 I don't really understand why the result is so different. I got similar result when I build the cyrstal server without --release flag, did you build it from make file? https://github.com/tbrand/which_is_the_fastest/pull/10

Or which version of crystal and llvm did you use?

[schovi]

@OvermindDL1 PHP vs ruby is pretty unfair. Don't forget roda and sinatra are still pretty large libraries in contrast to 4 line PHP file :) Can you try https://github.com/codeguy/Slim (found via google "PHP sinatra like frameworks") or something similar.

[tbrand]

Thanks!!! If you want additional discussion, please open new issue.:+1:

[OvermindDL1]

I got similar result when I build the cyrstal server without --release flag, did you build it from make file?

From the makefile yes.

Or which version of crystal and llvm did you use?

╰─➤  crystal --version
Crystal 0.22.0 [3c71228] (2017-04-20) LLVM 3.5.0

It should be the latest in the repositories from the offical site? I don't really use crystal so I just rely on it being updated that way. :-)

@OvermindDL1 PHP vs ruby is pretty unfair. Don't forget roda and sinatra are still pretty large libraries in contrast to 4 line PHP file :) Can you try https://github.com/codeguy/Slim (found via google "PHP sinatra like frameworks") or something similar.

Hence why I did not turn it in to a PR, it was just for my own morbid curiosity, and since the readme says 'as fast as possible'... ^.^

If you want I can set up a 'proper' test with that slim framework and PR it in, or just post the results here if you are curious?

[OvermindDL1]

I got a few minutes free here.

@schovi or anytone else if curious, I changed my basic PHP test to use slim instead (holy hell that thing is huge) and got these results (ordered from fastest to slowest, top to bottom again):

Language (Runtime)        Framework (Middleware)          Max [sec]       Min [sec]       Ave [sec]
------------------------- ------------------------- --------------- --------------- ---------------
rust                      iron                             4.648752        4.546433        4.597737
elixir                    plug                             5.498771        5.376086        5.452387
rust                      nickel                           5.570673        5.506968        5.541427
elixir                    phoenix                          6.101516        5.942020        6.042626
rust                      rocket                           6.786611        6.330680        6.452137
crystal                   router_cr                        8.769548        8.018409        8.528628
crystal                   kemal                           12.281298       10.394767       11.516996
php                       slim                            76.066147       74.935423       75.512605

The php file is:

<?php
require 'vendor/autoload.php';

$app = new Slim\App();

$app->get('/', function ($request, $response, $args) {
    return $response->write("");
});

$app->post('/user', function ($request, $response, $args) {
    return $response->write("");
});

$app->get('/user/{id}', function ($request, $response, $args) {
    return $response->write($args['id']);
});

$app->run();

?>

Definitely slowed down a lot. 😆

Elixir Phoenix is definitely the full-stack Rails-like contraption. Elixir Plug is the base-most simple router and such helpers thing. Elixir holds its own very well. :-)

And I confirmed that crystal was compiled with --release, it is consistently slower than elixir and rust as seen above on my system (AMD Phenom ][, 6 native cores, 3.6ghz, 16 gigs ram, linux).

[tbrand]

@OvermindDL1 Thanks for the valuable comment!! I still get a similar result (Crystal is faster than Rust and Elixir) on my Mac, so I'll try to take a benchmark on another server.

Anyway, such a feedbacks are very welcome!!:smile:

[OvermindDL1]

Anyway, such a feedbacks are very welcome!!😄

It is fun! Plus I am extremely curious why the crystal times were so different for me than on the chart, it makes me think I'm doing something wrong, but don't know what... :-(

I'm wondering if it is just because my system has more physical cores, which the rust and elixir versions can take advantage of better. Maybe I should run them restricted to 2 cores, hmm...

[OvermindDL1]

And here are the results with crystal, rust, and elixir with them constrained to only the first 2 out of the 6 physical cores, in top-to-bottom as fastest-to-slowest order as usual:

Language (Runtime)        Framework (Middleware)          Max [sec]       Min [sec]       Ave [sec]
------------------------- ------------------------- --------------- --------------- ---------------
crystal                   router_cr                        5.083894        4.410376        4.807896
rust                      nickel                           5.622460        5.601752        5.616635
crystal                   kemal                            7.244543        6.778164        6.980066
rust                      rocket                           7.212035        6.947049        7.023690
rust                      iron                             7.732281        7.595761        7.671830
elixir                    plug                            12.045980       11.741010       11.872830
elixir                    phoenix                         13.847535       13.547675       13.637995

Running them again while allowing them to use all 6 cores returns them to their prior speeds and orders with crystal being the slowest again, which is slower than the 2-core test, so I'm thinking that crystal is not multi-threaded well because I know that both Rust and the EVM (that Elixir runs on) are both able to be threaded crazy-well (with the EVM being almost linearly parallizable so it should eventually win above all with 'enough' cores).

Hmm, wish I could test on my 24 core server right quick, unable to any time soon though...

But yeah, testing on a comparatively tiny Mac computer (which really, no one would ever host servers on) is definitely not where the tests should be run. ^.^

[kelvinst]

Hmm, wish I could test on my 24 core server right quick, unable to any time soon though...

That would be reeeeeally interesting!!!!!!! 😮