Real World frameworks flag

[cyrusmsk]

Can we create some real world frameworks flag? To see frameworks that really possible to use in production, because some of the frameworks are just 'not real' and can be used only for simple benchmark games.

[waghanza]

Hi @cyrusmsk ,

I understand that it could be accurate to add a production-ready flag to this list.

Is that what you expect ?

[cyrusmsk]

Yes

[akyoto]

In the Go community, we have a convention of using version 0.x to denote an unstable API and 1.x or higher to indicate a stable API. I think this is widely adopted in most communities and could potentially help you implement this.

It's worth noting that "stable API" and "production ready" are not equivalent terms - you can have a production ready framework with an unstable programming interface, and vice versa. However, in general people use 1.x releases to indicate that a library is meant to be ready for the real world.

"Production ready" can also differ depending on who you ask. Take a micro framework that follows the UNIX philosophy and only implements routing and does that one job extremely well vs. a full-blown 200 dependencies project with every possible feature on the planet. This can be a highly subjective topic.

Therefore my suggestion is to use a metric that is not open to debate: Version numbers. Sure, you will have outliers who don't follow the convention at all, but in general the version number can give you at least an idea of the real world viability.

[waghanza]

Thanks for the suggestion @akyoto . The version of each framework is already shown on results

I'm not sure that we could define that ourselves by a simple rule.

I see the value of this info, but not sure we should have the responsability of displaying it.

[akyoto]

It's also worth noting that if this was implemented, you'd place the responsibility of evaluating a library for a use case out of the viewer's hands into your own hands. And that is not only playing with fire, it's also near impossible given the wide range of use cases out there in the real world.

The version numbers are already displayed and anything 0.x should be considered in development. Whether the status of a library is good enough for use case X should be something the viewer decides. I think it's very dangerous to attempt judging every single framework in this regard.

To clarify, I am against the idea of a flag that indicates real-world viability for every possible use case.

[waghanza]

I'm also opposed to.

The core value of this project is to expose some facts about technologies (frameworks) and how factually it can compete.

• How it performs ?
• Is it energy efficient ?
• Which budget should I use ?

The notion of real world usage is more like a subjective / adaptive field depending of context beyond technical aspects.

Thus it is not this project's responsibility to have such an info.

[cyrusmsk]

If the solution is very performant and energy efficient but only for this simplest benchmark, and completely useless for any real usage, why this information could be interesting?

[waghanza]

the scenario will not stay so simple ... @cyrusmsk

[akyoto]

I mean, there actually is one pattern where I would say "this passes the test but is useless for all developers, 100%":

Use of a switch-case block as your "router".

This is from the TechEmpower benchmarks and the fact that this is allowed there is absolutely incomprehensible.

[waghanza]

indeed, actual implementation is useless, we think of an implementation that mimic :

• external calls (external api, database, or anything that can be used remotely)
• a little bit of compute
• json serialization / deserialization

[akyoto]

we think of an implementation that mimic :

* external calls (external api, database, or anything that can be used remotely)
* a little bit of compute
* json serialization / deserialization

I see a couple problems with these. Let's break these down:

* external calls (external api, database, or anything that can be used remotely)

This is out of the scope for a micro framework. You might as well test sleep(x).

* json serialization / deserialization

This is also out of the scope for a micro framework. Why would I dictate which JSON library you have to use?

* a little bit of compute

This is the best one so far. The only criticism I can think of is that this will measure compiler performance more than framework performance. You will see all frameworks of a particular language reach equal numbers. The compute task needs to be so simple that it doesn't become the main factor you are benchmarking.

It is very easy to fall into the trap of benchmarking the wrong thing.

If you want to equalize the load across all languages, your only choice is a neutral factor like sleep(x). And even that is strictly speaking not equal because timer precision can differ across languages.

I like the "little bit of compute" idea or a sleep(x) for differing durations (no sleep / 1 ms / 10 ms / 100 ms / 1s) to emulate waiting for database replies. This way you avoid "library seeking hell" where every author just tries to find the fastest library for JSON / DB queries to pass your benchmark which measures external libraries and has nothing to do with the framework itself.

[akyoto]

I apologize for the double post, but I just remembered that 5 years ago I've already proposed a solution that would make it more real world like. And it wasn't my own idea, Julien Schmidt did it in his router benchmark where he loaded up ~200 routes from a routes definition file for every single framework. This loading of 200 routes from a text file fills up the routing tree, can be implemented in any language and would severely improve the quality of the benchmark data.

[waghanza]

What I have in mind when I talk about

external calls (external api, database, or anything that can be used remotely) is indeed a sleep 😛

for json, seems that this format is pretty standard a in a wide majority of language built-in. as the idea is to benchmark framework, we should use what is widely adopted, but in order not to make some unfaire benchmark, we should use 1 lib per language, either built-in or the widely adopted lib