Compare against using no framework at all and use it as base for the relative results

[motin]

This approach is used here: https://github.com/Big-Shark/forked-php-orm-benchmark#results and makes sense because what most really want to know is how much slower your app becomes by using a framework vs plain php.

This would make it easier to compare against different benchmarking environments as well. Also, it would make it less important whether or not phalcon.so was loaded or not.

[kenjis]

What do you mean by barebones php?

[motin]

No framework at all. Basically just uncomment https://github.com/kenjis/php-framework-benchmark/blob/963f129a124bb6ec2f8c46d731aa2ee704b34cc6/list.sh#L4

[kenjis]

As you know, no-framework is just echoing Hello Word!. But all the frameworks do more than just echoing Hello Word!.

Do you think it makes sense to compare them? I doubt it.

If you compare them, can you know how much slower your app becomes by using a framework vs plain php? Absolutely no. It is impossible! Because hello world is not your app.

This benchmarking tries to measure minimum overhead (minimum bootstrap cost) of PHP frameworks in the real world.

[motin]

Do you think it makes sense to compare them? I doubt it.

It makes much more sense to compare all the frameworks against the least common denominator, ie a PHP request that produces some output, than to have the relative numbers be against whichever framework that happens to be fastest during the test.

If you compare them, can you know how much slower your app becomes by using a framework vs plain php? Absolutely no. It is impossible! Because hello world is not your app. This benchmarking tries to measure minimum overhead (minimum bootstrap cost) of PHP frameworks in the real world.

Of course "Hello World" is not your app. However, the minimum bootstrap cost of PHP frameworks is NOT what is being measured in the benchmark currently. It also includes the cost of the request being made, the PHP interpreter to be loaded in memory and output being transferred to the client. All these things affect the throughput, memory and response time. However, by measuring the throughput, memory and the response time of "No framework", we get to know the results that are completely based on the hardware/OS/stack and NOT including any bootstrap cost of a PHP framework. By comparing this "base level" result and the results of a framework, we get the actual minimum bootstrap cost of that framework.

This makes it possible to get an idea of the minimum bootstrap cost also of the fastest performing framework, which otherwise just gets the relative performance values of "1.0".

It also makes the relative numbers more comparable across different benchmarks where for some reason the same frameworks could not be tested (like in the case of HHVM with Phalcon and Ice).

The "No framework" option does not however need to be listed alongside the other frameworks in case you believe that causes confusion, all I am saying is that it makes sense to compare the "relative" numbers against the no-framework case.

[motin]

Here is en example comparison against no-framework, including reversed relative throughput results (#39):

|framework          |requests per second|relative|peak memory|relative|
|-------------------|------------------:|-------:|----------:|-------:|
|no-framework       |           1,304.82|   100.0|       0.22|     1.0|
|phalcon-2.0        |             685.55|    52.5|       0.27|     1.2|
|ice-1.0            |             630.68|    48.3|       0.26|     1.2|
|tipsy-0.10         |             774.88|    59.4|       0.32|     1.4|
|fatfree-3.5        |             447.52|    34.3|       0.43|     1.9|
|slim-2.6           |             627.34|    48.1|       0.48|     2.1|
|ci-3.0             |             101.57|     7.8|       0.43|     1.9|
|nofuss-1.2         |             216.13|    16.6|       0.59|     2.6|
|slim-3.0           |             441.98|    33.9|       0.62|     2.8|
|bear-1.0           |              42.33|     3.2|       0.77|     3.4|
|lumen-5.1          |             270.75|    20.7|       0.00|     0.0|
|ze-1.0             |             259.65|    19.9|       0.80|     3.6|
|radar-1.0-dev      |             249.12|    19.1|       0.71|     3.2|
|yii-2.0            |             259.94|    19.9|       1.36|     6.0|
|silex-1.3          |             329.12|    25.2|       0.00|     0.0|
|cygnite-1.3        |             116.15|     8.9|       0.76|     3.4|
|fuel-1.8-dev       |              55.94|     4.3|       0.71|     3.2|
|phpixie-3.2        |              77.97|     6.0|       1.30|     5.8|
|aura-2.0           |             121.80|     9.3|       0.90|     4.0|
|cake-3.1           |             147.86|    11.3|       0.00|     0.0|
|symfony-2.7        |              56.43|     4.3|       0.00|     0.0|
|laravel-5.1        |              88.32|     6.8|       0.00|     0.0|
|zf-2.5             |              18.91|     1.4|       3.00|    13.3|
|typo3f-3.0         |               0.20|     0.0|       0.00|     0.0|

[motin]

Please re-open since the issue is not fixed yet.

To re-cap, only by using a no-framework baseline can the actual performance of the frameworks be evaluated neutrally, without having to be dependent on which framework happens to be the fastest in a particular instance.

This is particularly relevant when running benchmarks on your own setup, since it may not be feasible to get phalcon/ice up and running. In that case, it will still be possible to get relevant relative numbers if a comparison to no-framework is included.