Difficult to understand the ranks

[tbrand]

I love the expandable tables, however, the rankings are difficult to understand to me. As a first impression, "average" is higher priority than "50% percentile" since it is located left of it. But the ranks are sorted by latter one. It is very confusing.

Also I prefer to add numbers of ranks even if it is a bit verbose. Like this.

No. 1 👑 framework name No. 2 🌟the other framework No. 3 ⭐️ another one

I would like to use the table as an evidence, not ranking itself. So the above ranking should be detached from the evidence table.

What we want to tell? I would like to keep a which_is_the_fastest concept. (Not be a generic benchmark I hope.)

[tbrand]

Additionally, current tables are not ranking but just sorting results.

[waghanza]

@tbrand I understand what you mean when you speak abount columns, as lettre columns should have more valable data. Lets list columns that we want to display (that's thé easy part ^^^)

However, I do not understand the differences you make between ranking and sorting. The first element of each table is on top of the podium for me

[tbrand]

See the old commit. https://github.com/the-benchmarker/web-frameworks/tree/c890d049a878c108214fe185c4e623712e8d4dff

The ranking is very simple, detached from the detailed table and numbered. It's quite understandable for me.

Currently, there are too many frameworks to be listed all of them. So it might be good to list only top 10 frameworks.

[aichholzer]

In overall, the numbers are very inconsistent and mostly wrong...

[tbrand]

We'll try to improve the stability. Let me focus on how to show the result here.

[waghanza]

@tbrand I understand ranking is valuable. However ranking differs depending on point of view (what matters depending on usage). For example, rust is the number 1 language :stuck_out_tongue: but with actix-web for a req/s point of view, and with nickel for a latency point of view.

Also, performance of a real-world application will also vary depending on architecture in use. For me a so simple ranking is counter-productive, because of how we scale.

I mean if we display a simple list of more performing framework, it will largely be wrong => Having one iron backed app on 1 server could be less perming than 3 hanami app (in global depending of app usage).

I'm not keen on ranking (simple one) like *tfb** admin, btw, but if it could help ... I'm not opposed to

@aichholzer We now that results are inconsistent, for now. Btw, can you give use your point of view when you say they are wrong ? Wrong in face of what ?

[tbrand]

@waghanza

You should understand not only the rankings but also core value of this repository. People don't put stars for the profiling but for the rankings at first.

However ranking differs depending on point of view (what matters depending on usage)

We can fix the point of view. And don't have to show the language rankings now.

Also, performance of a real-world application will also vary depending on architecture in use. For me a so simple ranking is counter-productive, because of how we scale.

Then we just put a description for it on README.md. It's completely contradictory if we cannot make a rankings for a server. Then the profiling table means nothing as well.

Also please open another issue for the discussion of correctness if you want.

[waghanza]

@tbrand I know figures are not correct and can not be yet, I'll open an issue later if necessary :heart:

Honestly, I can not correlate the fact that people are putting stars with any conclusion, just that they appreciate our work

For me the first thing we must do, before anything (to avoid those question), is to define our personal goals with this project

What do you want to initialy solve, when you starters this project ?

[tbrand]

@waghanza

What do you want to initialy solve, when you starters this project ?

Show simple rankings of each framework/middleware for the same condition. The tables are just for evidence, not a result itself.

I would be discouraged if you guys misunderstand or change the core value...

[waghanza]

@tbrand The initial intention for contributing to this project was to help me to take a decision -> choose a stack (bunch of technologies) for a new (or a rewrite) of any project

My personal feeling / way to choose a tech
P	Passion
E	Evolution
P	Performance	The value of this project
S	Security

for me the next big steps are :

• creating production-ready results
• publish contents (blog or so) to explain
  • motivation
  • ecosystem
  • how to "read" results ?

What do you think about that ?

[waghanza]

I'll be off (holidays) for 3 weeks. I'll be glad to read you ideas :stuck_out_tongue_closed_eyes:

[tbrand]

The choosing tech motivation is good. But that is not a reason for not showing a ranking that I said.

Creating some other components (blog or some) is good for me. But that is not a reason for not showing a ranking that I said.

[waghanza]

Of course, I have no reason not to show ranking, we just need to find how 😎

[tbrand]

OK great.

How about limiting a number of frameworks to show on the ranking? There are too many frameworks to show now.

Like this. (Showing top 10 frameworks in this example)

Ranking of latency
1. brabra (lang)
2. hogehoge (lang)
3. fugafuga (lang)
...
10. fugefuge (lang)

Ranking of number of requests per sencond
1. brabra (lang)
2. hogehoge (lang)
3. fugafuga (lang)
...
10. fugefuge (lang)

All Results (Expandable table here)

[waghanza]

Ok. Even if I'm not fan, it could be interesting for someones.

I think 10 will be huge (having a Very long README).

How about ranking 3 or 5 langs, and the more performant framework per lang ?

[tbrand]

Yeah sure. 5 is acceptable. Could you work on it when you have times?

[waghanza]

sure, will do so for the next release (a release with only fix)

[foucist]

How about removing the 50% column?

http://latencytipoftheday.blogspot.com/2014/06/latencytipoftheday-most-page-loads.html https://bravenewgeek.com/everything-you-know-about-latency-is-wrong/

[waghanza]

Hi @foucist,

Thanks for participating in our project.

The links below, as I have understand them, is about using 50th percentile as a metric on ONE application context.

We are comparing several application with the same metrics, si we are not in the exact samedi context.

We can also decide to show 67th percentile ... The purpose of this issue is to define (understandable) columns to show, whatesoever is 50th or 67th or an other random number 😉

[waghanza]

I think an accurate percentile to show (amoung others) depends on the context.

For example, if I have a 2 servers behind a loas balancer, the more accurate is a 50th percentile sorting. The more optimized framework (for 2 backend servers) will then appear on top.

Samely, a 33th percentile will be accurate to use, if 3 backend servers are used.

I have to precise that percentile calculation only affect latency table.

[tbrand]

I think I could close the issue since @waghanza working on this at https://github.com/the-benchmarker/web-frameworks/pull/330