poor AB bench results - Githubissues

lubosdz commented 4 years ago

Hi,

not sure where to ask - so just shortly here:

I did apache bench on yii3 (demo) against yii2 (basic app) and got extremly poor results for yii3:

yii3 - only about 0.18 page per sec, 1 request takes about 5-6 secs
yii2 - OK, about 140 pages per sec

Using latest version of yii3 dev (composer update), PHP 7.4.1.

bench results here (must have switched to -n 1 due to extremly slow response):

Yii3


d:\wamp\bin\apache\apache2.4.37\bin>ab -n 1 -c 1 127.0.0.1/yii3/yii-demo/public/index.php
This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient).....done

Server Software:        Apache/2.4.37
Server Hostname:        127.0.0.1
Server Port:            80

Document Path:          /yii3/yii-demo/public/index.php
Document Length:        43 bytes

Concurrency Level:      1
Time taken for tests:   5.987 seconds
Complete requests:      1
Failed requests:        0
Non-2xx responses:      1
Total transferred:      344 bytes
HTML transferred:       43 bytes
Requests per second:    0.18 [#/sec] (mean)                    <- why? edit: found reason: 404 timeout
Time per request:       5986.504 [ms] (mean)
Time per request:       5986.504 [ms] (mean, across all concurrent requests)
Transfer rate:          0.06 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:  5987 5987   0.0   5987    5987
Waiting:      687  687   0.0    687     687
Total:       5987 5987   0.0   5987    5987

Is there anything special to configure for yii3 demo CLI environment .. ? In HTTP mode yii3 demo works all OK.

As for yii2 results are OK on the same setup:

Yii2

d:\wamp\bin\apache\apache2.4.37\bin>ab -n 1000 -c 1 127.0.0.1/yii2/basic/web/index.php
This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests

Server Software:        Apache/2.4.37
Server Hostname:        127.0.0.1
Server Port:            80

Document Path:          /yii2/basic/web/index.php
Document Length:        65901 bytes

Concurrency Level:      1
Time taken for tests:   6.571 seconds
Complete requests:      1000
Failed requests:        0
Non-2xx responses:      1000
Total transferred:      66116000 bytes
HTML transferred:       65901000 bytes
Requests per second:    152.20 [#/sec] (mean)
Time per request:       6.571 [ms] (mean)
Time per request:       6.571 [ms] (mean, across all concurrent requests)
Transfer rate:          9826.70 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   1.3      0      10
Processing:     0    6   4.7      9      16
Waiting:        0    6   4.7      7      16
Total:          0    6   4.8     10      16

Percentage of the requests served within a certain time (ms)
  50%     10
  66%     10
  75%     10
  80%     10
  90%     10
  95%     13
  98%     15
  99%     15
 100%     16 (longest request)

samdark commented 4 years ago

Is there anything special to configure for yii3 demo CLI environment .. ? In HTTP mode yii3 demo works all OK.

What do you mean by "HTTP mode" and "CLI environment"?

yiiliveext commented 4 years ago

At least yii2-basic doesn't have db queries on page load.

lubosdz commented 4 years ago

HTTP mode = browsing in a browser CLI mode = running command line shell (shellscript in windows), console

OK - will check what config is loaded in CLI and check DB connection.

Edit: slightly updated results for yii2 after running composer update.

lubosdz commented 4 years ago

After investigation I found:

Yii3 demo public/index.php - does not use DB connection. Even if it would, there is already pre-configured DB (for demo blog) using local SQLite, which does not require credentials. So DB connection in console is not an issue here.
there is exception throw in both console/web mode No definition for FastRoute\RouteParser at CompositeContainer.php line 32 . Silently ignored by configuration. This is not the reason, just wondering whether this is correct. Exceptions should not control request workflow.
returned document length 43 bytes - it simply returns 404 not found after 7 secs of timeout. But I cannot figure out what file cannot be found :-( Path to public/index.php is correct.

I am wondering - does AB bench in yii3-demo work well for anyone else?

Aside notes:

yii3 is much-much harder to track down issues than yii2. Learning curve is significantly steeper. Some parts feel overengineered in yii3. Even config files implement some kind of business logic.
there is lot of room for performance improvements - see profiling screen bellow.
memory usage shows 3-4x more than in yii2 (cca 1.5 MB against 4.5 MB). Actually while debugging yii3-demo I measured 22 MB memory usage (peak).

screen-yii3-demo-profile

samdark commented 4 years ago

@lubosdz top performance eaters are https://github.com/yiisoft/yii-demo/blob/master/public/index.php#L15

samdark commented 4 years ago

Also we haven't worked on performance yet focusing on design mostly.

yiiliveext commented 4 years ago

After investigation I found:

Yii3 demo public/index.php - does not use DB connection. Even if it would, there is already pre-configured DB (for demo blog) using local SQLite, which does not require credentials. So DB connection in console is not an issue here.

You should take into account the Cycle ORM initialization. Try to disable config rebuild and Cycle ORM.

samdark commented 4 years ago

Another thing is that debugger is turned on collecting ~500kb of data each request. https://github.com/yiisoft/yii-demo/blob/master/config/params.php#L8

shurik2k5 commented 4 years ago

Try to disable this: https://github.com/yiisoft/yii-demo/blob/a7595dbffb89a236c89e0e23a904eefbf47e6a5f/public/index.php#L15

samdark commented 4 years ago

Anyway, it should not take 6! seconds to respond and you can see it when running web app. There must be something that makes ab unhappy/fail.

terabytesoftw commented 4 years ago

Yii3 (yiisoft/yii-demo) AB benchmark http://demo.yiipeople.site:

[x] disable rebuild container.
[x] disable yii-debug.

[root@localhost php]# ab -n 1000 -c 1 http://demo.yiipeople.site/
This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking demo.yiipeople.site (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests

Server Software:        LiteSpeed
Server Hostname:        demo.yiipeople.site
Server Port:            80

Document Path:          /
Document Length:        4627 bytes

Concurrency Level:      1
Time taken for tests:   4.580 seconds
Complete requests:      1000
Failed requests:        109
   (Connect: 0, Receive: 0, Length: 109, Exceptions: 0)
Write errors:           0
Total transferred:      4914881 bytes
HTML transferred:       4626881 bytes
Requests per second:    218.33 [#/sec] (mean)
Time per request:       4.580 [ms] (mean)
Time per request:       4.580 [ms] (mean, across all concurrent requests)
Transfer rate:          1047.92 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:     4    4   0.4      4      10
Waiting:        4    4   0.4      4       9
Total:          4    5   0.4      4      10

Percentage of the requests served within a certain time (ms)
  50%      4
  66%      5
  75%      5
  80%      5
  90%      5
  95%      5
  98%      6
  99%      7
 100%     10 (longest request)

terabytesoftw commented 4 years ago

Yii3 (yiisoft/app) AB benchmark http://yiipeople.site:

[x] disable rebuild container.
[x] disable yii-debug.

[root@localhost php]# ab -n 1000 -c 1 http://yiipeople.site/
This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking yiipeople.site (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests

Server Software:        LiteSpeed
Server Hostname:        yiipeople.site
Server Port:            80

Document Path:          /
Document Length:        4162 bytes

Concurrency Level:      1
Time taken for tests:   3.752 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      4445000 bytes
HTML transferred:       4162000 bytes
Requests per second:    266.49 [#/sec] (mean)
Time per request:       3.752 [ms] (mean)
Time per request:       3.752 [ms] (mean, across all concurrent requests)
Transfer rate:          1156.80 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:     3    4   0.6      4      13
Waiting:        3    3   0.6      3      12
Total:          3    4   0.6      4      13

Percentage of the requests served within a certain time (ms)
  50%      4
  66%      4
  75%      4
  80%      4
  90%      4
  95%      4
  98%      5
  99%      5
 100%     13 (longest request)

terabytesoftw commented 4 years ago

Yii2 (yiisoft/app-basic) AB benchmark http://basic.yiipeople.site/index.php:

[root@localhost php]# ab -n 1000 -c 1 http://basic.yiipeople.site/index.php
This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking basic.yiipeople.site (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests

Server Software:        LiteSpeed
Server Hostname:        basic.yiipeople.site
Server Port:            80

Document Path:          /index.php
Document Length:        4107 bytes

Concurrency Level:      1
Time taken for tests:   4.391 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      4647000 bytes
HTML transferred:       4107000 bytes
Requests per second:    227.75 [#/sec] (mean)
Time per request:       4.391 [ms] (mean)
Time per request:       4.391 [ms] (mean, across all concurrent requests)
Transfer rate:          1033.57 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:     3    4   8.5      3     130
Waiting:        3    4   8.5      3     130
Total:          3    4   8.5      3     130

Percentage of the requests served within a certain time (ms)
  50%      3
  66%      3
  75%      3
  80%      4
  90%      4
  95%      4
  98%      5
  99%     68
 100%    130 (longest request)

samdark commented 4 years ago

Interesting results, @terabytesoftw. Have you turned off YII_DEBUG in Yii 2?

terabytesoftw commented 4 years ago

Interesting results, @terabytesoftw. Have you turned off YII_DEBUG in Yii 2?

Yes.

samdark commented 4 years ago

Then, despite not optimizing Yii 3 yet, it gives 266 rps vs 227 rps with Yii 2. That's unexpected but exciting.

lubosdz commented 4 years ago

Thanx for advices. Disabling Builder::rebuild() in index.php and yii-debug mode in config reduced memory usage from 4,5 MB to cca 1.5 MB - nice. But now the page rendering timer shows the number of secs since session start instead of page generation time e.g. Time: 402.6691 s. Memory: 1.2735 mb..

The 404 timeout in ab yii3-demo - still cannot figure out. But if ab it works on http://demo.yiipeople.site/ (linux) then my guess it could be related to windows 10 e.g. permissions or something. Will take deeper look into it next days.

EDIT: I am getting suspicious about .htaccess in yii3-demo config because:

when I access http://localhost/yii3/yii-demo/public/index.php - shows error We were unable to find the page /index.php. - this would explain 404 timeout in apache benchmark.
when I access http://localhost/yii3/yii-demo/public/ - (without specifying index.php) everything works OK

In yii2-basic accessing http://localhost/yii2/yii-basic/web/index.php (with or without index.php) works both OK.

Though I dont see anything wrong in .htaccess. I played with it e.g. adding Options +FollowSymLinks and mod_deflate - did not help so far.

alexey-egoshin commented 4 years ago

Guys, we very strong wait Yii3 and it must be faster than Yii2 of course! )

rossaddison commented 2 years ago

Working nicely. xdebug in apache's php.ini disabled as mentioned in your suggestions. yii-debug and api disabled. Significant speed increase.

Alternating between the following in config/params.php especially the first or former option ie. PhpFileSchemaProvider has increased performance significantly since syncing is mainly for building the database. The database is relatively secure so this first option is my preferred choice.

           \Yiisoft\Yii\Cycle\Schema\Provider\PhpFileSchemaProvider::class => [
               'mode' => \Yiisoft\Yii\Cycle\Schema\Provider\PhpFileSchemaProvider::MODE_READ_AND_WRITE,
               'file' => 'runtime/schema.php',
            ],

            \Yiisoft\Yii\Cycle\Schema\Provider\FromConveyorSchemaProvider::class => [
                'generators' => [
                     Cycle\Schema\Generator\SyncTables::class, // sync table changes to database
                ],
            ],

kmergen commented 1 year ago

I have the same issue ca. 5 sec per Request. but in chrome dev tools i have nearly the same good load results than in yii2-basic-app. (Load finish in ca. 100 ms, DomContentLoaded ca. 65 ms) I run yii3 application template on windows 11 with xampp and php 8.1 and apache 2.4.56.

Tried with env prod , debug false, without bootstrap(), comment out all unnecessary stuff, but always the same bad results.

samdark commented 1 year ago

That should be something about how ab is measuring time. We need to look into it but since real app responds in a timely manner to the browser, that's not a critical thing.

yiisoft / demo

poor AB bench results #124

Yii3

Yii2