search

kenjis / php-framework-benchmark

PHP Framework Benchmark
1.03k stars 204 forks source link

Switching the order of the benchmarks or running individually is inconsistent #47

Closed niallbrown closed 8 years ago

niallbrown commented 8 years ago

It maybe my VM setup but I can come close to your output when running the standard benchmarks out of the box. However, changing the order of the tests seems to affect the outcome dramatically as does running each one individually. If for example I run Tipsy or Laravel first I get results like >900 rps for them but changing them in the order drops by over 50% performance. Although I am running this with nginx at the moment I suspect the testing is not accurate at all since this change should not show such dramatic difference.

When running these individually I get a similar result to placing them at the top of the list, so I am wondering what part of the AB test is causing this or possible the scripting somehow??

nginx PHP 5.6.16-3+deb.sury.org~trusty+1 ubuntu 14.04

framework requests per second relative peak memory relative
laravel-5.1 933.13 8.3 0.00 0.0
laravel-5.2 559.62 5.0 0.00 0.0
fatfree-3.5 564.48 5.0 0.00 0.0
slim-2.6 520.47 4.6 0.00 0.0
ci-3.0 501.95 4.4 0.00 0.0
nofuss-1.2 466.65 4.1 0.00 0.0
slim-3.0 469.68 4.2 0.00 0.0
bear-1.0 408.57 3.6 0.00 0.0
lumen-5.1 404.35 3.6 0.00 0.0
ze-1.0 352.31 3.1 0.00 0.0
radar-1.0-dev 374.23 3.3 0.00 0.0
yii-2.0 112.95 1.0 0.00 0.0
silex-1.3 351.49 3.1 0.00 0.0
cygnite-1.3 351.01 3.1 0.00 0.0
fuel-1.8-dev 333.24 3.0 0.00 0.0
phpixie-3.2 327.72 2.9 0.00 0.0
symfony-2.7 320.02 2.8 0.00 0.0
zf-2.5 307.39 2.7 0.00 0.0
typo3f-3.0 291.83 2.6 0.00 0.0
phalcon-2.0 1,757.50 15.6 0.00 0.0
ice-1.0 1,833.87 16.2 0.00 0.0
framework requests per second relative peak memory relative
lumen-5.1 1,038.13 2.0 0.00 0.0
laravel-5.2 600.02 1.2 0.00 0.0
laravel-5.1 566.68 1.1 0.00 0.0
slim-3.0 551.69 1.1 0.00 0.0
symfony-2.7 514.67 1.0 0.00 0.0
framework requests per second relative peak memory relative
laravel-5.2 968.84 1.9 0.00 0.0
laravel-5.1 593.43 1.2 0.00 0.0
slim-3.0 576.18 1.2 0.00 0.0
lumen-5.1 531.08 1.1 0.00 0.0
symfony-2.7 499.94 1.0 0.00 0.0

Some others for comparison

framework requests per second relative peak memory relative
silex-1.3 345.10 3.0 0.00 0.0
laravel-5.1 559.34 4.9 0.00 0.0
laravel-5.2 545.42 4.8 0.00 0.0
fatfree-3.5 551.04 4.8 0.00 0.0
slim-2.6 517.77 4.5 0.00 0.0
ci-3.0 488.31 4.3 0.00 0.0
nofuss-1.2 468.78 4.1 0.00 0.0
slim-3.0 444.96 3.9 0.00 0.0
bear-1.0 420.13 3.7 0.00 0.0
lumen-5.1 396.25 3.5 0.00 0.0
ze-1.0 383.21 3.4 0.00 0.0
radar-1.0-dev 371.82 3.3 0.00 0.0
yii-2.0 114.07 1.0 0.00 0.0
cygnite-1.3 359.93 3.2 0.00 0.0
fuel-1.8-dev 342.59 3.0 0.00 0.0
phpixie-3.2 333.53 2.9 0.00 0.0
symfony-2.7 310.56 2.7 0.00 0.0
zf-2.5 319.69 2.8 0.00 0.0
typo3f-3.0 304.33 2.7 0.00 0.0
phalcon-2.0 2,018.81 17.7 0.00 0.0
ice-1.0 1,816.72 15.9 0.00 0.0
framework requests per second relative peak memory relative
laravel-5.2 968.75 8.2 0.00 0.0
laravel-5.1 593.21 5.0 0.00 0.0
slim-2.6 585.63 5.0 0.00 0.0
ci-3.0 528.41 4.5 0.00 0.0
slim-3.0 521.56 4.4 0.00 0.0
lumen-5.1 473.79 4.0 0.00 0.0
yii-2.0 118.21 1.0 0.00 0.0
silex-1.3 436.37 3.7 0.00 0.0
fuel-1.8-dev 429.55 3.6 0.00 0.0
symfony-2.7 393.60 3.3 0.00 0.0
zf-2.5 392.15 3.3 0.00 0.0
typo3f-3.0 384.68 3.3 0.00 0.0
phalcon-2.0 2,013.04 17.0 0.00 0.0
ice-1.0 2,051.81 17.4 0.00 0.0
niallbrown commented 8 years ago

Rebuilding the server and correcting some issues including moving to on disk from a share on the VM seems to have stabilized but the numbers are still off for some of the frameworks. Will try testing some other configurations.