search

the-benchmarker / web-frameworks

Which is the fastest web framework?
MIT License
6.91k stars 641 forks source link

Clarify results #281

Closed waghanza closed 5 years ago

waghanza commented 5 years ago

Hi,

Following the #238, it COULD be awesome to report :

It also COULD help clarify results. May developers argue that comparing so languages such as node and some such as elixir (which targets are completely different) are not very relevant.

This PR show results order by req/s AND by latency

Regards,

waghanza commented 5 years ago

@OvermindDL1 Should not we use percentile, instead of average latency to rank by latency ? it will be more accurate no ?

OvermindDL1 commented 5 years ago

50th percentile, 99.99th percentile, and standard deviation would be the best to show as it shows the latency range and how it grows.

waghanza commented 5 years ago

@OvermindDL1 my question is want metric (one ?) use to order latency tab ?

is it accurate to differentiate both table ?

OvermindDL1 commented 5 years ago

I'd include all information on both tables, but I'd use two table, one sorted on req/s, the other sorted on 50th percentile latency.

If only Github had the ability to mark a table as sort-by-click-on-header. Could always make a dedicated HTML for that though (github pages or so).

waghanza commented 5 years ago

Don't you think tables will be huge ?

My final purpose is to compute result, store them in a BigQuery or else, and having a website displaying them

@tbrand @OvermindDL1 What do you think ?

OvermindDL1 commented 5 years ago

Eh, the tables would be long sure, but that seems fine to me (especially if you can clamp the headers to the top as it scrolls).

waghanza commented 5 years ago

@OvermindDL1 On a race between ruby / node / php / python, I have different results (ranks) for latency :

I suggest then to keep existing version (sorting by average) and adjust if after using a cloud to compute results

OvermindDL1 commented 5 years ago

Low latencies at high percentiles most likely means they generate little to no garbage in comparison. :-)

waghanza commented 5 years ago

@OvermindDL1 not understood

OvermindDL1 commented 5 years ago

Just me guessing at what might cause 50 and 99+ percentile differences, ignore. ^.^

For note, this is a good chart for how to display the overarching information:

Add in req/sec as well and it would be a good detail showing for each framework.

image

tbrand commented 5 years ago

Could you remove the column which is not related to each ranking? For example, remove Requests / s for Ranked by latency ranking.

Other things are acceptable for me :+1:

waghanza commented 5 years ago

I've done a little change to the benchmarker code base, i know use a redis to store result (redis was picked because of simplicity but it add a dependency).

We need to refactor using an other key/value store (to avoid adding dependency) between :

OvermindDL1 commented 5 years ago

(redis was picked because of simplicity but it add a dependency)

As long as it's ran via docker or so and not needed to be installed. Why not just use leveldb/rockdb if it's just a KV store needed or sqlite for querying? Or just map?

waghanza commented 5 years ago

@OvermindDL1 I mean a redis installed on sieger machines, not targets. I need a k/v store to store all results as json (at least in order to let us decide what informations to render in each tables)

OvermindDL1 commented 5 years ago

I need a k/v store to store all results as json (at least in order to let us decide what informations to render in each tables)

Definitely no point in using something as crazy-heavy as redis, just embed leveldb or rocksdb, I use both rather excessively, and they will far far outperform redis.

waghanza commented 5 years ago

@OvermindDL1 sure, it was I have said :stuck_out_tongue: the list above was essentially for me picking an idea ^^

'cause it was late last night (and I was not very productive), I take a random tool -> I'll rewrite this part

waghanza commented 5 years ago

@OvermindDL1 The idea with this modification was to decide what to show in tables, I have made for this a sample here => https://github.com/waghanza/which_is_the_fastest/blob/clarify_results/README.md#results but I'm not very convinced

@tbrand What do you think ?

PS : I have taken advantage of this, by adding a warm-up call :stuck_out_tongue:

waghanza commented 5 years ago

@OvermindDL1 for me ranking by latency SHOULD be ranked by 50th perc. (more reliable -> ex : rack-routing SHOULD the 5th cause of 50th perc., it is actually the 2th 'cause of average)

@georgeu2000 do you observe same results ?

OvermindDL1 commented 5 years ago

@waghanza yep, 50% makes sense, for someone else then 99% may make sense, etc... etc... I really wish github had sortable column by headers, maybe need to set up a github pages so they become clickable. ^.^;

waghanza commented 5 years ago

@OvermindDL1 I have find that it's possible, but very hard to use / maintain (sortable table). my idea is to push results is BigQuery or else and create a website for this (github's page when #252 :stuck_out_tongue: )

OvermindDL1 commented 5 years ago

No need for BigQuery, just generate an html file via a template and use some progressively enhancing javascript to make the table sortable via column.

waghanza commented 5 years ago

@OvermindDL1 I know it could be done like that, but I find it mor flexible to separate compuation from layout

e.g -> use travis to compute, and a website to display results :stuck_out_tongue_winking_eye:

moreover, compute and generate markdown on travis need to push from travis to github, no a big fan of this kind of solution, but I suggest we open an other issue for this question :stuck_out_tongue_closed_eyes:

waghanza commented 5 years ago

@tbrand Just after merging this, I'll release :100: