Why my Lumen is so much slower than Taylor's?

[kenjis]

According to http://taylorotwell.com/how-lumen-is-benchmarked/ :

framework	requests per second
Slim 2	1700-1800
Lumen	1800-1900
Slim 3	1200-1300

• Lumen is a bit faster than Slim2.

But my benchmarks (2015/04/15) :

framework	requests per second
slim-2.6	774.31
lumen-5.0	372.07

• Slim2 is about two times faster than Lumen.

I've found some differences in code, but overall benchmarking methodology seems to be the same.

don’t be surprised if your numbers are different than my numbers.

Yes, benchmarking environments change numbers. But the difference (relatively to Slim2) is too much, isn't it?

I simply want to know why. What makes the huge difference?

[kenjis]

My VM had small memory, so I changed it to 2GB.

framework	requests per second	relative	peak memory	relative
slim-2.6	827.37	18.4	0.50	1.0
yii-2.0	413.81	9.2	1.75	3.5
silex-1.2	364.08	8.1	1.00	2.0
lumen-5.0	381.07	8.5	1.00	2.0

But still slim2 is much faster.

[neomerx]

@kenjis

• kernel settings
• PHP engine
• testing script & the way you use ab

When I researched Laravel I started with the same kernel settings as you. However I found out I had performance results degraded rapidly after each ab run and therefore were totally unreliable. I'm not saying I changed them to best possible but I did have more stable results.

Taylor didn't disclosed what PHP engine was used but most likely he had chosen HHVM for the reason it better optimizes Laravel/Lumen than it's competitors (from my experience). I think mostly for 'code inlining' feature.

Try testing only a couple frameworks at the same time (e.g. slim vs lumen only) and run them manually with delays and more than one time. That's my numbers how Laravel reacts on a few sequential ab runs on HHVM 374.73, 682.83, 584.68, 771.33, 729.74, 734.95 RPS.

[samdark]

So, questions:

1. Which engine was used? HHVM or Zend PHP?
2. Which version of engine?
3. Which PHP settings?
4. Was APC/OpCache used?

[samdark]

Also, ab isn't really good way to test. siege is much better.

[barryvdh]

I think Taylor said that he didn't test with HHVM, and he used Homstead (Vagrant): http://laravel.com/docs/5.0/homestead#included-software So Nginx + php5.6.

Maybe try running the benchmarks from within Homestead?

[neomerx]

@barryvdh As I understand it Homestead is just Ubuntu + VirtualBox. Thus adding HHVM to it is just running in terminal something like

sudo apt-get install software-properties-common && sudo apt-key adv --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 0x5a16e7281be7a449 && sudo add-apt-repository 'deb http://dl.hhvm.com/ubuntu trusty main' && sudo apt-get update && sudo apt-get install hhvm && sudo service nginx restart

[samdark]

Then it depends very much on which OS is running Vagrant, which VM (WMWare, Virtualbox) and what CPU runs it. In my experience such VM-tests were too far from being accurate so I was setting up a clean 10$ box at DigitalOcean.

[harikt]

@samdark +1 for using DO :) .

[kenjis]

@samdark

My environment:

• CentOS 6.5 64bit (VM)
  • PHP 5.5.23 (remi RPM)
  • Zend OPcache v7.0.4-dev
  • Apache 2.2

php.ini is probably default of remi.

such VM-tests were too far from being accurate

Yes, comparing to your real production environment. And I encourage all of you run benchmarks on your environment for your purpose.

But do relative numbers change so big? In VM, A is twice faster than B, but at DigitalOcean A is as fast as B. Could it be? I don't know.

[FractalizeR]

@kenjis I think it's worth to publish this configuration (OS + PHP) in README so that people know what php version and opcache were used.

Also PHP 5.5 will enter EOL on 20 Jun. May be we should move to 5.6 in tests?

It would be also interesting to benchmark using HHVM.

[kenjis]

@neomerx @barryvdh Thank you for your comments.

First I'll try Homstead later. It may give me some clues.

[FractalizeR]

I think also, that tests must be done in a real OS, not in the VM. For example, VirtualBox at my MacOS (Intel Core i7 2.4GHz, 8Gb RAM) with Laravel gives me page rendering time at about 600ms. That is partly because of slow vbox virtual file system.

[neomerx]

@FractalizeR What about moving to SSD from floppy? Kidding you )

[fesor]

@FractalizeR, VirtualBox FS is not so slow. Maybe you trying to run apps stored in shared directory (vboxsf is terribly slow)?

[samdark]

But do relative numbers change so big? In VM, A is twice faster than B, but at DigitalOcean A is as fast as B. Could it be? I don't know.

Yes. That's what I was experiening while testing Yii 2.0 HHVM vs PHP 5.6. In my case on VM HHVM was slower than Zend Engine.

[barryvdh]

Yeah but relatively speaking, if framework A is 2x as fast as framework B, it should probably provide the same relative results in other tests, unless there are major differences in disk i/o, cpu intensive tasks, memory exhaustion or different bottlenecks.

It would be interesting to see what those differences would be, so the frameworks could be optimised perhaps.

[samdark]

Correct. VM could be such bottleneck.

[harikt]

If anyone need to make the fastest micro framework, you MUST use FastRoute for routing. Nothing else can break it ;) . The performance gain is mostly on routing.

[FractalizeR]

VirtualBox FS is not so slow. Maybe you trying to run apps stored in shared directory (vboxsf is terribly slow)?

@fesor Yep, that's exactly what I do. vboxsf is terribly, terribly slow.

[neomerx]

@harikt I like the idea of having different implementations for routing and nikic without any doubts is a brilliant developer however Laravel and Lumen have exactly the same routing system with drastic performance difference. So other parts of the system are the biggest problems. You probably get some gain for many routes with FastRoute but how many routes are there in default applications? 1-3. I wouldn't expect much of using it in this race.

[barryvdh]

@neomerx Lumen is using nikic/fast-route, Laravel is using Symfony routing.

[barryvdh]

See this thread for more detailed info by Taylor: http://www.reddit.com/r/PHP/comments/32kajb/lumen_php_microframework_by_laravel/

[neomerx]

@barryvdh Right. I meant piping actually.

[neomerx]

I suggested Taylor using FastRoute for L5 and it's good to see it's finally there )

[harikt]

@neomerx FastRoute miss some of the good things like domain routing of symfony. So fast doesn't means featured. We may need to use the right router to do the work.

[neomerx]

@harikt FastRoute is about implementing this interface

interface Dispatcher {
    const NOT_FOUND = 0;
    const FOUND = 1;
    const METHOD_NOT_ALLOWED = 2;
    /**
     * Dispatches against the provided HTTP method verb and URI.
     *
     * Returns array with one of the following formats:
     *
     *     [self::NOT_FOUND]
     *     [self::METHOD_NOT_ALLOWED, ['GET', 'OTHER_ALLOWED_METHODS']]
     *     [self::FOUND, $handler, ['varName' => 'value', ...]]
     *
     * @param string $httpMethod
     * @param string $uri
     *
     * @return array
     */
    public function dispatch($httpMethod, $uri);
}

You're free to support whatever you want

[seyfer]

Some answers here http://stackoverflow.com/questions/29647960/difference-and-similarities-between-lumen-and-laravel/29648574#29648574

Some facades was commented..

[kenjis]

@FractalizeR I added. https://github.com/kenjis/php-framework-benchmark#benchmarking-environment

[FractalizeR]

@kenjis Great, thanks.

[fesor]

Is it possible to create Docker container for every example? I think this would be match better to support.

Also, maybe we should warmup opcache/app caches with preflight request right before run ab?

[FractalizeR]

I am not sure we need docker here, but I definitely second opcache warmup.

[harikt]

@neomerx agree that we can implement ourself. Then may need to benchmark with that featured version.

Edit : Corrected the sentence so it doesn't mean you can implement yourself.

[koriym]

FYI @kenjis Here's another great showcase how to benchmark helloworld app. https://github.com/pmjones/php-framework-benchmarks

[christoferd]

Benchmarking hello-world app should be "just for fun", I would not rely on those numbers to help me make choices.

Laravel will slow down a lot after growing your code with; ORM, ORM helper magic methods, and Blade

ORM - a lot of stuff is automated, they use string parsing to work out stuff for you, including pluralisation

ORM magic - again, automated stuff to make your life easier, ++ string parsing

Blade - string parsing!

What you will need is knowledge on how to speed it up; Cache, fast HDD, a lot of RAM, etc.

Benchmarking should be done after first load, where a raw first load benchmark is useless, as after cache kicks in the numbers are much different. Artisan Optimize also changes a lot.

IMO Laravel = Convenience, if you are looking for speed, look at different frameworks like phalcon. I believe Laravel allows us to code projects quicker. IMO any/most project can start with Laravel, small medium business that affords it can also afford a rebuild later in future as they scale up, where that rebuild may be in another framework to suit the newer requirements.

PS: I would love to know how a very large Laravel application with thousands of users can survive on a server? how fast does it run at peak use? and how that server is decked out? That would be interesting and educational.

All the best,

Chris | RandomCodes.com Product & Services Manager

Ph: +61 (0) 3 5294 0297 | Mob: +61 (0) 419 571 596 | Skype: chris.dedavid RandomCodes.com http://randomcodes.com/ | Quality Features http://randomcodes.com/doc/?doc=Quality_Features&ref=email_sig | Order Document http://randomcodes.com/doc/?doc=Order_Form&ref=email_sig ...at your service.

.-. .- -. -.. --- -- -.-. --- -.. . ... .-.-.- -.-. --- --

On 18 April 2015 at 00:33, Akihito Koriyama notifications@github.com wrote:

FYI @kenjis https://github.com/kenjis Here's another great showcase how to benchmark helloworld app. https://github.com/pmjones/php-framework-benchmarks

— Reply to this email directly or view it on GitHub https://github.com/kenjis/php-framework-benchmark/issues/13#issuecomment-93997650 .

[ghost]

I guess the reason is @taylorotwell has SSD (and uses MacOS as host os) and @kenjis has HDD (and uses other OS as host).

[Ardakilic]

@plakhin Taylor uses Virtualbox + Vagrant + Homestead, which is a Linux virtual machine. Even though, the relative results should not differ that much.

[christoferd]

@Arda Most Mac machines have SSD or they have HDD in RAID and operate faster than average non Mac PC.

The HDD setup would make a big difference. Eg. I have 3 x 80mb per second drives in RAID 0.... I get 240mb/p/s throughput. That's much faster than average.

My other RAID is SSD and runs over 400mbps. So, I could run benchmark using "Virtualbox + Vagrant + Homestead" and get different results.

PS: Garbage collection settings would also affect things sometimes, should turn it off in benchmarks. Or better yet, run the tests many times to get an average result.

• Chris

On 20 April 2015 at 21:47, Arda Kılıçdağı notifications@github.com wrote:

@plakhin https://github.com/plakhin Taylor uses Virtualbox + Vagrant + Homestead, which is a Linux virtual machine. Even though, the relative results should not differ that much.

— Reply to this email directly or view it on GitHub https://github.com/kenjis/php-framework-benchmark/issues/13#issuecomment-94431221 .

[ghost]

@Ardakilic I mentioned MacOS as a HOST OS, which works directly with disk drive. I'm sure you get very different results testing drive i/o performance even in the same virtual environment and nearly same disk drive but on MacOS vs Linux vs Windows as a host OS.

[Ardakilic]

@christoferd Indeed, but would that change the "relative results" ?

[kenjis]

Homestead (Vagrant):

• Ubuntu 14.10
  • PHP 5.6.7
  • Zend OPcache v7.0.4-dev
  • Nginx 1.6.2

framework	requests per second	relative	peak memory	relative
slim-2.6	966.62	2.3	0.50	0.0
silex-1.2	424.03	1.0	0.00	0.0
lumen-5.0	463.34	1.1	0.00	0.0

I don't know why but silex and lumen do not output memory usage: https://github.com/kenjis/php-framework-benchmark/blob/master/lumen-5.0/public/index.php#L30

Note to Homestead setup: I installed ab command and php-intl after vagrant up. And I changed Nginx config a bit to work my code. I disabled xdebug.

server {
    listen 80;
    server_name homestead.app;
    root "/home/vagrant/public";

    index index.html index.htm index.php;

    charset utf-8;

    location = /favicon.ico { access_log off; log_not_found off; }
    location = /robots.txt  { access_log off; log_not_found off; }

    access_log off;
    error_log  /var/log/nginx/homestead.app-error.log error;

    sendfile off;

    client_max_body_size 100m;

     location ~ [^/]\.php(/|$) {
        fastcgi_split_path_info ^(.+\.php)(/.*)$;
        fastcgi_param PATH_INFO $fastcgi_path_info;
        fastcgi_pass unix:/var/run/php5-fpm.sock;
        fastcgi_index index.php;
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_intercept_errors off;
        fastcgi_buffer_size 16k;
        fastcgi_buffers 4 16k;
    }

    location ~ /\.ht {
        deny all;
    }
}

[kenjis]

My Host is MBA 2013, 1.3GHz Intel Core i5, SSD 128GB, Mac OS X 10.10.3.

[ghost]

@Ardakilic The answer is obvious, Lumen uses disk drive much more (I don't think that's good, but we are not talking about it at this time) so it's results are depends on disk system much more.

[ghost]

@kenjis 128 GB SSD is twice slower than 256 GB in same 2013 MBA and ~4 times slower than in MBPro.

Can anyone run the test on 2013-2015 SSD MB PRO or at least on 256 GB MBA 2013-2105 ?

[kenjis]

vagrant@homestead:~$ sudo hdparm -t /dev/mapper/vagrant--vg-root

/dev/mapper/vagrant--vg-root:
 Timing buffered disk reads: 1068 MB in  3.00 seconds = 355.88 MB/sec
vagrant@homestead:~$ sudo hdparm -t /dev/mapper/vagrant--vg-root

/dev/mapper/vagrant--vg-root:
 Timing buffered disk reads: 1094 MB in  3.01 seconds = 364.01 MB/sec
vagrant@homestead:~$ sudo hdparm -t /dev/mapper/vagrant--vg-root

/dev/mapper/vagrant--vg-root:
 Timing buffered disk reads: 1104 MB in  3.00 seconds = 367.47 MB/sec

[kenjis]

For your information.

My benchmarking environment (VirtualBox; CentOS 6.5):

[vagrant@localhost ~]$ sudo hdparm -t /dev/mapper/VolGroup-lv_root

/dev/mapper/VolGroup-lv_root:
 Timing buffered disk reads: 706 MB in  3.00 seconds = 235.26 MB/sec
[vagrant@localhost ~]$ sudo hdparm -t /dev/mapper/VolGroup-lv_root

/dev/mapper/VolGroup-lv_root:
 Timing buffered disk reads: 708 MB in  3.00 seconds = 235.84 MB/sec
[vagrant@localhost ~]$ sudo hdparm -t /dev/mapper/VolGroup-lv_root

/dev/mapper/VolGroup-lv_root:
 Timing buffered disk reads: 692 MB in  3.00 seconds = 230.52 MB/sec

[neomerx]

Maybe mount Lumen folder to RAM and mitigate HDD/SSD influence? How big this folder, 20Mb?

# fstab
tmpfs /path/to/lumen tmpfs defaults,noexec,nosuid 0 0

[zandzpider]

(No VM)

Ran tests on Macbook Pro mid 2014 2.2Ghz i7 16GB ram 250 GB SSD on PCI-Express bus

php --version PHP 5.6.7 (cli) (built: Apr 21 2015 10:06:35) Copyright (c) 1997-2015 The PHP Group Zend Engine v2.6.0, Copyright (c) 1998-2015 Zend Technologies

curl -IL http://127.0.0.1:80 HTTP/1.1 200 OK Server: nginx/1.6.3

ab -t 10 -c 10 http://lumen.app/

Requests per second: 645.35 #/sec

[barryvdh]

@zandzpider it's not about the RPS in absolute value, but relative to the other frameworks.

[defenestrator]

I am fascinated by these performance discussions about web frameworks, especially PHP frameworks. In the real world too much of the internet is beyond our control, so all these suppositions and assertions inevitably fail when released into the wild.

I wonder how many of those in this discussion will ever build something that requires scalability beyond what any of the above-mentioned frameworks are capable of.

If you need extreme high-speed, web-scale, non-blocking, deeply concurrent, fault-tolerant, highly portable rock-star tech, then PHP is not a great solution. In the event you really need something like that; Go learn Go. (Er learn Erlang.) Heck you could get into NodeJS and see improved performance over PHP. If one's project needs to scale to the size of Twitter, for instance, then you have a good and fortunate problem to solve! But until then this is all intellectual wanking. I'm not trying to piddle on the parade, I just want to keep that fact in focus. There are better languages if you're a speed demon.

[samdark]

PHP scales well horizontally because of share-nothing nature. Look at the projects made with it: Yahoo, Facebook, Wikipedia, Flickr, Digg, SourceForge, VK.com, mailchimp etc.