Most of real-life frameworks use I/O layers. Shouldn't they be here as well?

[mmoreram]

I really love this project. If you're new in the web environment and you want to choose a framework because of performance, then this can be your list. You can check a language and look for a project that fits your needs.

But nowadays, this might be a not as good representation as we could think in these terms.

Mostly everyone is used to working with I/O in web frameworks. Let's say a database, Redis or just filesystem layers. By adding a small (set/get) layer here by using, for example, a tiny PostgreSQL db, I'm sure that the list would be sorted in another way. I see lots of frameworks that just are representing a single file, while in real life, this is not something that it works. Instead of that, having this layer, would strongly empower these projects that lead with resources better.

[waghanza]

In confess the implementation is not the panacea. As the list of framework because every day longer (or at least regularly) the only caveat is find (if we decide to change) is to take time to imagine an implementation that reflect nowadays server-side API needs.

It will be a pleasure, if you open a discussion on this subject. Their is already some issues about that, but we can reasonably say that within a certain amount of time & reflexion we should decide and start to change implementations.

[mmoreram]

@waghanza it will depend mostly on what message this project wants to have within the list.

For example, and taking by example base they are both 2 frameworks that I really know a lot, (Important. This is just an example. Long life to Symfony)

• Symfony
• DriftPHP

DriftPHP is built on top of reactive promises. In real life, projects are used to needing a server in front of the front controller. Drift doesn't need it. It works asynchronously, based on non-blocking I/O calls and, on top of that, can use several of the bundles from the Symfony ecosystem. You can create a small API with a database connection, and handle thousands of requests per second by using a single PHP thread. No matter how long these database queries last. Furthermore, DriftPHP reuses the same services among all requests, with no memory leaks.

In Symfony, this doesn't happen. The slowest the DDBB query is, the less requests per second you can achieve. One kernel per request and need several nginx workers (CPUs) to achieve the same numbers than Drift in terms of requests per second with no I/O operations.

In this list, Symfony has more requests / second than DriftPHP.

In this example, than can be surely done with other frameworks like DriftPHP and other frameworks like Symfony, the message is... "In terms of performance, Symfony is better than Drift". And this is not true. At least by using the same resources and emulating a real life scenario.

Here some tips about what in real life really matters.

• An entity PUT (You test how the FW access to request data and build internal HTTP domain infrastructure). You test as well more heavy database access (writes) and how the whole project deals with it. - Frameworks dealing properly with fire & forget will take advantage of that
• An entity GET. Simple. Just get 1 entity without using cache at all. Fastest I/O or asynchronous frameworks will take advantage of that
• A simple HEAD. Frameworks working with reusable infrastructure and domain, like reusable kernels, will take advantage of that
• A simple GET with some heavy CPU usage, The thing here is not to block next requests when something is happening in the server. Non-blocking frameworks working with asynchronous processes will take advantage of that.

Of course, frameworks dealing with multi-threads, keep-alive connections, better connection management... will take advantage as well, like they are doing right now.

I don't think that we need more probes than these. Performance is a composition of these parts (for discussion).

Update

• First steps on that direction would basically be to provide a database with a small table with 2 or 3 fields. I would go for mysql as absolutely all web frameworks will have access to it.
• In terms of adding new endpoints, the project could check if a new route exists to check that is applying the new specs. You could maintain the old one, and for frameworks willing to test these new specs, add a new list.

WDYT?

[waghanza]

Two things :

• I prefer to avoid setup a database layer (I think some have already discussed about a sleep to mimic db ops)
• Symfony requires php/fpm + nginx to run. I can understand the results (indeed actually it is on top, but as long as this project will be mature, the disadvantages will be shown -> resources usages)
• Having 2 specs are too complex, but we can PR to add some routes

I'll open a discussion this week end to summarize what we said and wait some times, to start implement it

[mmoreram]

@waghanza yes, a small sleep would do the work as well :)

[ujibang]

A simple proposal to simulate actual work is requiring the web service to compute a bcrypt hash of a random string with a given complexity, e.g. 12 (the complexity can be tuned).

BCrypt is "slow by design" to avoid brute force attacks. This will put an hard computation task on the server without requiring to change the testing infrastructure.

wrk can send a random fixed length string in the request body, the web service will send back the bcrypted hash of the string. The wrk script can then validate then result (this should be computationally easy compared to the server-side hashing task)

From wikipedia: There are implementations of bcrypt for C, C++, C#, Elixir,[4] Go,[5] Java,[6][7] JavaScript,[8] Perl, PHP, Python,[9] Ruby, and other languages.

[mmoreram]

@ujibang using a hard computational action here will not do that work, basically because PHP is what it is, and will not change among all frameworks (only will improve from 7 to 8).

The point here is an emulation of an I/O operation, and that can be easily solved with an sleep function.

[ujibang]

@mmoreram, I think that the impact of a sleep() will be minimal for event loop based (or async) frameworks (as node.js). On the opposite, sleep() will hit multithread framework a lot.

I really think a blocking task will better simulate a real activity.

[mmoreram]

Exactly @ujibang, and that's the point, bringing some advantage to these frameworks working on top of non-blocking operations, instead of just returning a simple Response.

Adding a blocking task that makes the same impact for everyone will not change at all the final result.

[medz]

A simple proposal to simulate actual work is requiring the web service to compute a bcrypt hash of a random string with a given complexity, e.g. 12 (the complexity can be tuned).

BCrypt is "slow by design" to avoid brute force attacks. This will put an hard computation task on the server without requiring to change the testing infrastructure.

wrk can send a random fixed length string in the request body, the web service will send back the bcrypted hash of the string. The wrk script can then validate then result (this should be computationally easy compared to the server-side hashing task)

From wikipedia: There are implementations of bcrypt for C, C++, C#, Elixir,[4] Go,[5] Java,[6][7] JavaScript,[8] Perl, PHP, Python,[9] Ruby, and other languages.

@ujibang I think your idea is detrimental to the performance of web frameworks.

First of all, the performance of cryptographic calculations such as BCrypt is not controlled by the web framework. Instead, it is limited by language performance.

Web frameworks should focus more on:

1. Routing performance (including: constant path segment, variable path segment, catchall, etc.)
2. Template rendering performance (if available, some full stack web frameworks will have it)
3. request data processing performance
4. response body data response performance

[waghanza]

and 5 simulate database calls (by sleep or other). the idea is to mimic what happens IRL.

indeed @mmoreram it could be the detriment of async frameworks as drift, but I think this could be more relevant that a simple hello world, as we have now

[medz]

@mmoreram I disagree, most web frameworks themselves do not support database connections. Instead, integrate other third-party database ORM and other tools. Your idea should be as a new benchmark test project. Its function is to compare the IO performance of each ORM (or database connector) interacting with the database.

[medz]

Except for PHP, web frameworks in other languages ​​are relatively pure. Especially in languages that require AOT, their APIs are relatively low, and the web framework provides basic routing and request processing capabilities. There used to be countless projects in c/.net/rust/dart that tried to be all-encompassing, but ultimately failed (the projects became unmaintained because they were too big). For compiled languages, their types and other restrictions are stricter, and developers prefer to combine them by themselves.

[mmoreram]

@medz I disagree with your disagreement.

So a framework 100% blocking and a framework 100% async can be equally performant, then? I mean yes, we can make a performance ranking with all of them, and all of them can probably make the same number of requests per second, basically because without I/O, a framework is just a bunch of non-blocking serialized lines of code. A couple of foreaches can make one framework 1ms faster than the other one causing, instead of 15K reqs/s, do 12K reqs/s, but that's it. Useful? nah, at all (please, don't misunderstand me. I think that these kind of lists are important, and I appreciate so much the work done in this project). Because you will not find any framework without I/O anywhere, and part of the framework work is to provide a fast and performant I/O gates to install any ORM, DBAL, etc. This is part of the framework as well, not only these foreaches (For example, Symfony is sync, ReactPHP is async. Please, take a look at possible infrastructure costs for both techs with same traffic. It is insane how much one scales and how much the other does. Not only with request/response times, reactPHP can work with microseconds, but how persistent connections can work on both sides, inmemory cache between requests, etc).

Yes. We can write this ranking list, and of course all frameworks can run to simplify the number of foreaches (yes, I know, I'm oversimplifying maybe too much. Sorry), but if someone is trying to find out performant frameworks for production, that will be fake data. Won't be real.

I'm using ReactPHP and DriftPHP in all my company's service, and in my life I've experimented this kind of performance in PHP, Python or Java. Never. This should be represented in this list.

[medz]

@mmoreram

Sorry, I understand what you're thinking, because I once dedicated five years to PHP, but I haven't used it seriously in six years. Of course, I was also a laravel follower. I made a lot of contributions to laravel Chinese documentation in China to help them.

But what I want to tell you is that given your needs. You should not look at the benchmarks of the framework, but the comprehensive evaluation between the frameworks. Your situation almost always occurs in PHP (they are keen on making frameworks that cover everything (such as database, cache, view engine, etc.) .

I can't say they're wrong, as they're keen to provide developers with an environment where most development can be done without too many options. But what if we look beyond Java and PHP? The developers here are more pursuing "purity", and the things they develop have nothing extra. Web framework Everyone is optimizing routing performance and better HTTP protocol parsing capabilities. ORM is more interested in exploring connection reuse and exploring future models that are more in line with modern technology (such as Prisma).

I think the readme of this warehouse is very clear. Its purpose is to compare the benchmark performance of web frameworks. It needs to take into account all development languages and the frameworks under these languages.

Web frameworks in most languages do not come with any database ORM, no Cache support, no Redis, and no View. If tested the way you think, these frameworks would never be able to join the fray.

From this, the next the-benchmarker/web-frameworks will appear on GitHub, and this repository will lose any interest in joining it from authors who are more focused on a certain field.

I am the author of the Spry framework myself, and I am also the author of Prisma for Dart. If this happens, I think most frameworks like Spry will withdraw from this competition, because we are competing for request/response processing capabilities, and we are competing for whose routing is faster.

I would never let my two open source projects Spry and Prisma for Dart come together to fight someone else's project.

[medz]

Another thing I hope you learned is in languages like Dart/.net/C/C++/Rust/Zig. Our web framework authors will choose to develop HTTP protocol parsing ourselves and put it into the framework in order to optimize it at the 1 nanosecond level.

And I found that PHP/Node/Java are more keen to implement frameworks on their own http. You can see that in this benchmark test, almost all the ones that ranked very well implemented their own HTTP protocol parsing. At least in PHP I don't see anyone implementing HTTP protocol parsing in PHP and handling requests using Socket raw links.

Of course, this does not mean that no one is doing such things. I admire them very much when they are doing such things. They have received high stars, but no developers will actually use it in their own frameworks. Due to language limitations, such methods will cause some languages that are already limited by language performance to use it, resulting in very low performance.

To put it bluntly: a certain PHP framework did such a thing, its performance was very low and its ranking was very low.

I hope you understand the difference between "benchmark testing" and "comprehensive testing".

[medz]

Sorry, I think I have something to add.

I just exchanged the ideas you mentioned with several other developers offline. They had the same idea as me and told me another thing.

Yes, there is another repository that implements what you are talking about. But it starts with the same thing as the-benchmarker/web-frameworks and turns your attention to benchmarking.

But later, various IO tests and database tests were added. This has more requirements for web frameworks, and they require that a framework must have databases, redis and other things.

I think you should be able to search for the name of that warehouse (I don’t want to mention it). Its current behavior has made developers lose interest in continuing to contribute to it, so that the popularity of tokio in Rust can be said to have Rust is a top-notch high-performance IO framework that cannot be found in their repository or ranking website.

But I guess it should be what you want it to be. For web frameworks, it even tests database connection performance, processing power, and Redis connection performance. But is this what web frameworks are supposed to do? No, this is something that a full-stack framework can do.

Regarding it, I would say very rudely: "You don't deserve to be called a web framework benchmark. It should be renamed a comprehensive evaluation ranking of development frameworks."

[cyrusmsk]

@medz I think that the main criticism of that "full-stack with database" benchmark is related to the way of the code prepared. Many solutions are using outdated versions (because they are faster), some dirty hacks (which are no way to be used in any production ready scenario).

This benchmark is supposed to have other idea. Up-to-date versions of the environment and libraries, normal production code without hacks.

But I agree that "hello world" is not always the most desired information.. and real usage of the frameworks could be different.

It will be interesting to find comments from real business who are running web projects. What are their main requirements for the solution? Latency? Cost of running server on AWS (usage of RAM/CPU)? % of lost connections? Also micro-service architecture (when you sending JSONs) vs Web application (with rich and heavy page with JS and data) could work differently for different frameworks.

It will be interesting to create some scenarios and metrics that could be used for framework efficiency estimation.

[medz]

@cyrusmsk Yes, that's very interesting. It can help enterprises better choose the technology stack. But as a web framework developer and an orm developer, I also developed a full-stack framework, but I don't want this repository to be like that. Because your idea can describe business performance, but it cannot correctly disclose framework performance. I still think it is more appropriate to create a new repo to do something similar than to destroy the current benchmark.

[waghanza]

I'm currently aligned with @cyrusmsk vision. I mean the idea of this benchmark is, as he said, to use up-to-date version / without hack / the most clean way ... the persistence layer will come (I mean will be mimic since having an ORM here will mess-up the results facing the idea of comparing frameworks)

the whole idea behind this project is to help dev/cto/whatever decide/choose a piece of tech based on facts, performance is only one approach

perhaps, I need to clarify it in readme