Laravel tests not realistic.

taylorotwell commented 9 years ago

You would need to run the following:

php artisan optimize --force. php artisan config:cache.

taylorotwell commented 9 years ago

After performing the optimizations on Laravel 5 (like you would in the real world), it clocks in slightly faster than Cake. Please update your tests or remove Laravel as it's not even remotely accurate since it is not installed properly.

kenjis commented 9 years ago

Thank you for your feedback! I will update my benchmarks.

vinkla commented 9 years ago

@kenjis Nice! Have you updated the README as well?

kenjis commented 9 years ago

I ran my benchmark, but current results is like below:

framework	requests per second	relative	peak memory	relative
cake-3.0	272.60	3.4	1.00	2.0
symfony-2.6	131.94	1.6	2.00	4.0
laravel-5.0	81.10	1.0	2.75	5.5

Laravel5 results improved slightly, but not faster than Cake. I don't know why @taylorotwell 's laravel is so fast. If you have any clue, would you please teach me?

neomerx commented 9 years ago

@taylorotwell @kenjis the result is realistic BUT Laravel out of the box comes with middleware such as 'maintenance check', 'cookie encryption', 'session', etc which is not really needed for this particular 'Hello world' test. When it switched off and reduced number of empty/not used providers I got

from 390 rps (L5) vs 3984 rps (codeigniter)
to 1514 rps (L5)

If Taylor add some optimizations to piping who knows L5 might get even more :+1: (session and cookie provider that could be really switched off could add even more)

samdark commented 9 years ago

That would leave you w/o framework, wouldn't it? :) Abstraction always costs extra performance, it's not a big surprise.

neomerx commented 9 years ago

@samdark Really not. You will have IoC, HTTP kernel pipeline, controllers, support for DB/cookies/cache/auth/bus/ and etc and even console artisan. Not all of them are needed for the test but cannot be switched off completely without side effects.

taylorotwell commented 9 years ago

I think this discussion basically proves how worthless these types of "benchmarks" are.

On Mon, Mar 30, 2015 at 4:13 AM, neomerx notifications@github.com wrote:

@samdark https://github.com/samdark Really not. You will have IoC, HTTP kernel pipeline, controllers, support for DB/cookies/cache/auth/bus/ and etc. Not all of them are really needed for the test but cannot be switched off completely without side effects.

— Reply to this email directly or view it on GitHub https://github.com/kenjis/php-framework-benchmark/issues/1#issuecomment-87605094 .

samdark commented 9 years ago

Not really. These benchmarks show how bare framework performs i.e. what's the overhead. While it matters mostly in cases when application code is perfectly optimal, it still matters. At least sometimes.

ConnorVG commented 9 years ago

I've got to agree with @taylorotwell on this one - it's a useless benchmark. You're not even close to comparing apples to apples here.

samdark commented 9 years ago

@ConnorVG why? Isn't it comparing typical middleware request cycle overhead?

neomerx commented 9 years ago

I've made a research (link below) and come to the following conclusions

Kernel stack, TCP stack, PHP and web server should be optimized to unlock the application potential.
Requests per second alone cannot be reliable measure for checking application performance. Specialized tools should be used instead.

Way more detailed explanation TL;DR :) https://github.com/neomerx/rhw-l5#speed-up-real-hello-world-application-laravel-5

samdark commented 9 years ago

Great post. Love how detailed it is and the conclusion.

Still, I think that the benchmark itself is OK since I doubt there are alike significant default template changes @kenjis made to any of frameworks tested. At least I can say so about Yii.

TheCelavi commented 9 years ago

@kenjis Don't get discouraged, these kind of benchmarks are far, far away from useless. I am going to state two, in my opinion, most important:

Lower bound for resources: these kind of benchmarks provide us with lower bound for resources requirements. With these kind of data, we can project in best case scenario which hardware we need for specific project depending on technology we use and do we need to scale in advance. It is not an ideal, but we have some kind of a picture in front of us (in comparison to nothing).
Motivation to improve: of course you are going to make some people angry - and that is cool, as long as they will be motivated to improve. Competition is always good and for us as end consumers of frameworks - we do not see anything wrong in making framework even more optimised. Per example, I am SF2 developer, SF2 is much better for development then horrific CI, but CI is much faster (of course it is, it has nothing in it). Speed is not the reason for me to switch to CI, god forbid, but why SF2 should not be as fast as CI? I would surely like to be so :)

Please, keep up with good work! And thank you for this data, this is great contribution to community!

neomerx commented 9 years ago

@TheCelavi you might need to know more about framework internals, their differences and got some experience with benchmarking. This test compares bicycles to tractors where some go down the hill and some go up. However I agree on motivation.

TheCelavi commented 9 years ago

@neomerx after all this years I still get surprised how people can stoop so low - you have never met me, you have never worked with me, you have no idea who I am, and nevertheless, you entitled yourself with a right to speculate about my education, knowledge and working experience. Let me return you the curtesy of speculating about what do you do - you are Laravel developer, right?

neomerx commented 9 years ago

@TheCelavi your statements were unjustified. You basically said 'Guys, listen, I want you to know my unjustified thoughts'. If you have anything meaningful to say welcome https://github.com/neomerx/rhw-l5#speed-up-real-hello-world-application-laravel-5

neomerx commented 9 years ago

@TheCelavi why supporting such tests is a bad idea https://github.com/kenjis/php-framework-benchmark/pull/5

TheCelavi commented 9 years ago

@neomerx I am sorry? What I said was not addressed to "some guys", nor to you, it was mentioned to @kenjis. I said thanks because I see my personal benefit from this, now I have some lover bound for SF2 (which I commonly use), some XY req/sec that can be max achieved - so I can do some rough estimations and projections what do I need, hardware wise, for some project release before I even start to work on it. It is not precise info, but it is a start, and I am ok with that. I am sorry that that kind of info does not have value for you - but it has a value for me. And that data is provided free of charge, so thanks @kenjis again. I would like to buy you a beer - and I am very serious about it.

Regarding second statement - well, I am sorry, from now on, when I recognise something that is good (by my personal opinion, of course) I will try not to step on your toe and publicly express my appreciation and gratitude, I will from now on send a personal note and/or email. This is my first time, so I really hope you are going to forgive me this one.

Regarding "bad idea to support this kind of tests" - if these kinds of tests are unrealistic and/or irrelevant and/or does not contain any value -> why would you waste your time here, why do you care what some stupid test say? It is stupid, right?

And after all - may I remind you that this is a free world, I can support whatever I want. I choose to support this. Go benchmark! Go! And?

kenjis commented 9 years ago

@TheCelavi Thank you!

I show you all how to benchmark, so you can freely interpret the results. And you can benchmark on your own environment and tweek them as you like or think worthy.

Regarding second statement - well, I am sorry, from now on, when I recognise something that is good (by my personal opinion, of course) I will try not to step on your toe and publicly express my appreciation and gratitude, I will from now on send a personal note and/or email. This is my first time, so I really hope you are going to forgive me this one.

What is "second statement"? The paragrah is for me or not? Sorry I'm not good at English.

By the way, I updated my benchmarks. Now 13 frameworks.

kenjis commented 9 years ago

I applied all artisan command: https://github.com/kenjis/php-framework-benchmark/blob/master/setup.sh#L63-65

But my Laravel is not as fast as Cake. That is my question.

barryvdh commented 9 years ago

Nice to see you added Lumen already, good job @kenjis

I tried the benchmarks on my own server (php5.4 + apc, centos, apache), which gave similar results as the results you posted (relatively speaking of course) and puts Lumen around the same rps as Yii 2 and Silex 1.2 (But not 90% fasteras Silex, as stated on the Lumen website, more like 20%).

But of course it all depends on the server you run the benchmarks on..

@taylorotwell Does Lumen need some more optimizing, or is it correctly configured as is?

TheCelavi commented 9 years ago

@kenjis It was response addressed to neomerx which disapproves my saying "thanks" and "I appreciate your efforts" to you in public manner.

taylorotwell commented 9 years ago

Ok these Lumen benchmarks just confirm my suspicions that these benchmarks are totally screwed up. Lumen is almost twice as fast as Silex and I can post reproducible bash scripts to demonstrate that.

On Wednesday, April 15, 2015, Nikola Svitlica notifications@github.com wrote:

@kenjis https://github.com/kenjis It was response addressed to neomerx which disapproves my saying "thanks" and "I appreciate your efforts" to you in public matter.

— Reply to this email directly or view it on GitHub https://github.com/kenjis/php-framework-benchmark/issues/1#issuecomment-93285863 .

samdark commented 9 years ago

Please do, it's interesting.

harikt commented 9 years ago

@taylorotwell I myself did one, but not sure Lumen is that fast as mentioned. I am also interested to see the same.

taylorotwell commented 9 years ago

I will literally post a video of my benchmarking these frameworks today to put this to rest. Including blog.

On Wednesday, April 15, 2015, Hari K T notifications@github.com wrote:

@taylorotwell https://github.com/taylorotwell I myself did one, but not sure Lumen is that fast as mentioned. I am also interested to see the same.

— Reply to this email directly or view it on GitHub https://github.com/kenjis/php-framework-benchmark/issues/1#issuecomment-93377662 .

TheCelavi commented 9 years ago

@taylorotwell Why not the code as well so we can reproduce your tests on our local machines?

samdark commented 9 years ago

:+1:

taylorotwell commented 9 years ago

Yes I will post the code because this is the most weird and misleading benchmarks I've ever seen and I'm sick of hearing about it.

On Wednesday, April 15, 2015, Alexander Makarov notifications@github.com wrote:

[image: :+1:]

— Reply to this email directly or view it on GitHub https://github.com/kenjis/php-framework-benchmark/issues/1#issuecomment-93379598 .

TheCelavi commented 9 years ago

g1391797501863903228

taylorotwell commented 9 years ago

http://taylorotwell.com/how-lumen-is-benchmarked/

neomerx commented 9 years ago

@taylorotwell why your browser shows port 8000 however you test ab with default port 80 (2:36)? Is it a homestead feature?

barryvdh commented 9 years ago

The benchmarks are probably running inside Homestead, on the default port 80. From outside, the port 8000 is mapped to 80 in homestead.

neomerx commented 9 years ago

@barryvdh Yep most likely. It just wasn't clear. Anyway even after L5 kicks I think the idea of comparing very different out-of-the-box functionality of frameworks and used methodology are less than optimal.

kenjis commented 9 years ago

@taylorotwell Thank you. But unfortunately it seems there is no secret recipe.

So I still don't know why my Laravel is not as fast as Cake.

I close this issue, because I accepted all your suggestions and discussion goes to Lumen. So I created new issue: https://github.com/kenjis/php-framework-benchmark/issues/13

My question will be solved if I know why my Lumen is so much slower than yours. If you guys have some clue or opinion, please post. I appreciate your posts.

jeteon commented 9 years ago

I commend the initiative of trying to get benchmarks like these out. Being able to have some comparison of PHP frameworks is really useful when one has a performance critical application. That being said, I think most people who find themselves with a demand that approaches the capacity of these frameworks, probably have the resources to just write straight, application specific PHP code, which will out-perform any framework. Even a "slow" 81 RPS is 6 998 400 unique hits per day, and that's just the ones that would be getting through your cache. And by that point, other things in the request chain will probably be playing a much bigger role in your slowest requests. I feel a lot of these benchmarks neglect to give you a feel for what "reasonable" is with these numbers.

However, given the obvious philosophical differences, with frameworks, I think its dangerous for people to walk away with the conclusion that it provides some baseline or "lower bound". Realistically, all a benchmark like this can (and should) ever provide is what you can expect out of a framework on day one, before you put in any of your own code. I think your primary concern at this point is getting your application started and all you actually need is maybe 4-5 RPS to support development. Some frameworks opt to give you everything you might need so that you can get going quickly and then trim things later when performance is more of a concern. Others give you almost nothing and have you build up what you need as you go. Either approach has pros and cons but taking the results of these benchmarks at face value without normalizing for this factor can only lead to questionable conclusions.

kenjis / php-framework-benchmark

Laravel tests not realistic. #1