search

treefrogframework / treefrog-framework

TreeFrog Framework : High-speed C++ MVC Framework for Web Application
https://www.treefrogframework.org
BSD 3-Clause "New" or "Revised" License
1.29k stars 220 forks source link

Performance low compare with tomcat, test tools apache ab #82

Closed hks2002 closed 7 years ago

hks2002 commented 8 years ago

I created one new app with tspawn, then use all default setting. Then use Apache Benchmark(ab) did a test, the result is as bellow: Treefrog: Requests per second: 9.73 #/sec Tomcat: Requests per second: 640.02 #/sec

How to solve it? My Treefrog is windows mingw version.

c:>ab -n 100 -c 100 http://127.0.0.1:8800/403.html This is ApacheBench, Version 2.3 <$Revision: 655654 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient).....done

Server Software: TreeFrog Server Hostname: 127.0.0.1 Server Port: 8800

Document Path: /403.html Document Length: 177 bytes

Concurrency Level: 100 Time taken for tests: 10.281 seconds Complete requests: 100 Failed requests: 0 Write errors: 0 Total transferred: 86500 bytes HTML transferred: 17700 bytes Requests per second: 9.73 #/sec Time per request: 10281.267 ms Time per request: 102.813 [ms](mean, across all concurrent requests) Transfer rate: 8.22 [Kbytes/sec] received

Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 1.6 0 16 Processing: 10094 10158 38.7 10156 10234 Waiting: 31 95 38.3 94 172 Total: 10094 10158 38.5 10156 10234

Percentage of the requests served within a certain time (ms) 50% 10156 66% 10172 75% 10188 80% 10203 90% 10219 95% 10219 98% 10234 99% 10234 100% 10234 (longest request)

c:>ab -n 100 -c 100 http://127.0.0.1:8080/docs/403.html This is ApacheBench, Version 2.3 <$Revision: 655654 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient).....done

Server Software: Server Hostname: 127.0.0.1 Server Port: 8080

Document Path: /docs/403.html Document Length: 673 bytes

Concurrency Level: 100 Time taken for tests: 0.156 seconds Complete requests: 100 Failed requests: 0 Write errors: 0 Total transferred: 88900 bytes HTML transferred: 67300 bytes Requests per second: 640.02 #/sec Time per request: 156.245 ms Time per request: 1.562 [ms](mean, across all concurrent requests) Transfer rate: 555.64 [Kbytes/sec] received

Connection Times (ms) min mean[+/-sd] median max Connect: 0 1 4.3 0 31 Processing: 16 72 45.2 94 141 Waiting: 0 69 47.1 94 141 Total: 16 73 45.0 94 141

Percentage of the requests served within a certain time (ms) 50% 94 66% 109 75% 109 80% 125 90% 125 95% 125 98% 141 99% 141 100% 141 (longest request)

c:>

treefrogframework commented 8 years ago

I think response of static contents is not fast, because it's application server. In that case, you could use Web server, nginx or apache.

http://www.treefrogframework.org/documents/view/showing-static-contents

..but too slow..

And, try this,

---application.ini--- MPM.thread.MaxAppServers=8 <-- here

proostas commented 8 years ago

I got interested in these ab results and gradually come to understand that it's something wrong with what ApacheBench does.

I've got simple controller which renderJson()'s QVariantMap on /test.

First, ab: stas@fehu:~# ab -n 100 -c 100 http://127.0.0.1:8800/test This is ApacheBench, Version 2.3 <$Revision: 1430300 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient).....done

Server Software: TreeFrog Server Hostname: 127.0.0.1 Server Port: 8800

Document Path: /test Document Length: 29 bytes

Concurrency Level: 100 Time taken for tests: 10.038 seconds Complete requests: 100 Failed requests: 0 Write errors: 0 Total transferred: 46500 bytes HTML transferred: 2900 bytes Requests per second: 9.96 #/sec Time per request: 10038.456 ms Time per request: 100.385 [ms](mean, across all concurrent requests) Transfer rate: 4.52 [Kbytes/sec] received

Connection Times (ms) min mean[+/-sd] median max Connect: 4 11 4.0 11 18 Processing: 10016 10017 1.0 10017 10019 Waiting: 0 2 1.2 2 4 Total: 10021 10028 4.9 10028 10037

Percentage of the requests served within a certain time (ms) 50% 10028 66% 10031 75% 10033 80% 10034 90% 10036 95% 10036 98% 10037 99% 10037 100% 10037 (longest request)

But, what happens here?: stas@fehu:~# ab -n 1 -c 1 http://127.0.0.1:8800/test This is ApacheBench, Version 2.3 <$Revision: 1430300 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient).....done

Server Software: TreeFrog Server Hostname: 127.0.0.1 Server Port: 8800

Document Path: /test Document Length: 29 bytes

Concurrency Level: 1 Time taken for tests: 10.017 seconds Complete requests: 1 Failed requests: 0 Write errors: 0 Total transferred: 465 bytes HTML transferred: 29 bytes Requests per second: 0.10 #/sec Time per request: 10017.252 ms Time per request: 10017.252 [ms](mean, across all concurrent requests) Transfer rate: 0.05 [Kbytes/sec] received

Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.0 0 0 Processing: 10017 10017 0.0 10017 10017 Waiting: 0 0 0.0 0 0 Total: 10017 10017 0.0 10017 10017

Next, I rechecked with other tools and got absolutely normal results.

curl -w "@curl-format.txt" -o /dev/null -s "http://localhost:8800/test" time_namelookup: 0.000 time_connect: 0.000 time_appconnect: 0.000 time_pretransfer: 0.000 time_redirect: 0.000 time_starttransfer: 0.001

    time_total: 0.001

stas@fehu:~# httperf --server=localhost --port=8800 --uri=/test --num-conns=10000 --num-calls=1 httperf --client=0/1 --server=localhost --port=8800 --uri=/test --send-buffer=4096 --recv-buffer=16384 --num-conns=10000 --num-calls=1 httperf: warning: open file limit > FD_SETSIZE; limiting max. # of open files to FD_SETSIZE Maximum connect burst length: 1

Total: connections 10000 requests 10000 replies 10000 test-duration 3.215 s

Connection rate: 3110.2 conn/s (0.3 ms/conn, <=1 concurrent connections) Connection time [ms]: min 0.3 avg 0.3 max 1.1 median 0.5 stddev 0.0 Connection time [ms]: connect 0.2 Connection length [replies/conn]: 1.000

Request rate: 3110.2 req/s (0.3 ms/req) Request size [B]: 66.0

Reply rate [replies/s]: min 0.0 avg 0.0 max 0.0 stddev 0.0 (0 samples) Reply time [ms]: response 0.2 transfer 0.0 Reply size [B]: header 436.0 content 29.0 footer 0.0 (total 465.0) Reply status: 1xx=0 2xx=10000 3xx=0 4xx=0 5xx=0

CPU time [s]: user 1.16 system 0.64 (user 36.1% system 20.0% total 56.1%) Net I/O: 1612.8 KB/s (13.2*10^6 bps)

Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0 Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

Can anyone elucidate these strange ab's results? I think it has something to do with some dodgy ab's benchmark algorithms.

proostas commented 8 years ago

Rechecked with harsher concurency: root@fehu:~# httperf --server=localhost --port=8800 --uri=/test --num-conns=10000 --num-calls=1 --rate 5000 httperf --client=0/1 --server=localhost --port=8800 --uri=/test --rate=5000 --send-buffer=4096 --recv-buffer=16384 --num-conns=10000 --num-calls=1 httperf: warning: open file limit > FD_SETSIZE; limiting max. # of open files to FD_SETSIZE Maximum connect burst length: 380

Total: connections 9561 requests 9561 replies 9561 test-duration 2.046 s

Connection rate: 4672.3 conn/s (0.2 ms/conn, <=1022 concurrent connections) Connection time [ms]: min 0.6 avg 188.3 max 361.3 median 204.5 stddev 48.2 Connection time [ms]: connect 65.4 Connection length [replies/conn]: 1.000

Request rate: 4672.3 req/s (0.2 ms/req) Request size [B]: 66.0

Reply rate [replies/s]: min 0.0 avg 0.0 max 0.0 stddev 0.0 (0 samples) Reply time [ms]: response 123.0 transfer 0.0 Reply size [B]: header 436.0 content 29.0 footer 0.0 (total 465.0) Reply status: 1xx=0 2xx=9561 3xx=0 4xx=0 5xx=0

CPU time [s]: user 0.09 system 0.37 (user 4.2% system 17.9% total 22.1%) Net I/O: 2422.8 KB/s (19.8*10^6 bps)

Errors: total 439 client-timo 0 socket-timo 0 connrefused 0 connreset 0 Errors: fd-unavail 439 addrunavail 0 ftab-full 0 other 0

And still got great results.

treefrogframework commented 8 years ago

I understand that the bottleneck of performance on TreeFrog is global locking, currently I am implementing so as to remove it. That's very hard in thread programing, but the performance would be improved more if its done successfully.

treefrogframework commented 7 years ago

Percentage of the requests served within a certain time (ms) 50% 10156 66% 10172 75% 10188 80% 10203 90% 10219 95% 10219 98% 10234 99% 10234

... 10 seconds. This is KeepAlive timeout.

Try this, application.ini

    -HttpKeepAliveTimeout=10
    +HttpKeepAliveTimeout=0
hks2002 commented 7 years ago

when use apche ab, add -k (keepalive) parameter, and application.ini HttpKeepAliveTimeout not set to 0, the result become good.

with -k and HttpKeepAliveTimeout=0 result: RPS=198

C:\Users\Administrator>ab -k -n 1000 -c 100 http://127.0.0.1:8800/mainpage This is ApacheBench, Version 2.3 <$Revision: 1757674 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient) Completed 100 requests Completed 200 requests Completed 300 requests Completed 400 requests Completed 500 requests Completed 600 requests Completed 700 requests Completed 800 requests Completed 900 requests Completed 1000 requests Finished 1000 requests

Server Software: TreeFrog Server Hostname: 127.0.0.1 Server Port: 8800

Document Path: /mainpage Document Length: 3254 bytes

Concurrency Level: 100 Time taken for tests: 5.047 seconds Complete requests: 1000 Failed requests: 0 Keep-Alive requests: 0 Total transferred: 3616000 bytes HTML transferred: 3254000 bytes Requests per second: 198.14 [#/sec] (mean) Time per request: 504.688 [ms] (mean) Time per request: 5.047 [ms] (mean, across all concurrent requests) Transfer rate: 699.69 [Kbytes/sec] received

Connection Times (ms) min mean[+/-sd] median max Connect: 0 4 43.0 0 516 Processing: 47 484 285.2 594 1141 Waiting: 16 425 239.1 547 875 Total: 47 488 285.0 594 1141

Percentage of the requests served within a certain time (ms) 50% 594 66% 625 75% 656 80% 688 90% 734 95% 1094 98% 1109 99% 1109 100% 1141 (longest request)

with -k and HttpKeepAliveTimeout=10 result: RPS=941

C:\Users\Administrator>ab -k -n 1000 -c 100 http://127.0.0.1:8800/mainpage This is ApacheBench, Version 2.3 <$Revision: 1757674 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient) Completed 100 requests Completed 200 requests Completed 300 requests Completed 400 requests Completed 500 requests Completed 600 requests Completed 700 requests Completed 800 requests Completed 900 requests Completed 1000 requests Finished 1000 requests

Server Software: TreeFrog Server Hostname: 127.0.0.1 Server Port: 8800

Document Path: /mainpage Document Length: 3254 bytes

Concurrency Level: 100 Time taken for tests: 1.063 seconds Complete requests: 1000 Failed requests: 0 Keep-Alive requests: 1000 Total transferred: 3639000 bytes HTML transferred: 3254000 bytes Requests per second: 941.17 [#/sec] (mean) Time per request: 106.250 [ms] (mean) Time per request: 1.063 [ms] (mean, across all concurrent requests) Transfer rate: 3344.66 [Kbytes/sec] received

Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.7 0 16 Processing: 0 103 275.5 16 1047 Waiting: 0 102 275.5 16 1047 Total: 0 103 275.6 16 1047

Percentage of the requests served within a certain time (ms) 50% 16 66% 16 75% 16 80% 31 90% 156 95% 1000 98% 1000 99% 1000 100% 1047 (longest request)

without -k and HttpKeepAliveTimeout=10 result: RPS=9

C:\Users\Administrator>ab -n 1000 -c 100 http://127.0.0.1:8800/mainpage This is ApacheBench, Version 2.3 <$Revision: 1757674 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient) Completed 100 requests Completed 200 requests Completed 300 requests Completed 400 requests Completed 500 requests Completed 600 requests Completed 700 requests Completed 800 requests Completed 900 requests Completed 1000 requests Finished 1000 requests

Server Software: TreeFrog Server Hostname: 127.0.0.1 Server Port: 8800

Document Path: /mainpage Document Length: 3254 bytes

Concurrency Level: 100 Time taken for tests: 100.091 seconds Complete requests: 1000 Failed requests: 0 Total transferred: 3639000 bytes HTML transferred: 3254000 bytes Requests per second: 9.99 [#/sec] (mean) Time per request: 10009.098 [ms] (mean) Time per request: 100.091 [ms] (mean, across all concurrent requests) Transfer rate: 35.50 [Kbytes/sec] received

Connection Times (ms) min mean[+/-sd] median max Connect: 0 1 16.4 0 516 Processing: 9422 9996 98.4 9999 10561 Waiting: 16 165 126.2 141 625 Total: 9422 9997 96.8 9999 10561

Percentage of the requests served within a certain time (ms) 50% 9999 66% 10015 75% 10031 80% 10046 90% 10061 95% 10063 98% 10092 99% 10515 100% 10561 (longest request)

without -k and HttpKeepAliveTimeout=0 result: RPS=171

C:\Users\Administrator>ab -n 1000 -c 100 http://127.0.0.1:8800/mainpage This is ApacheBench, Version 2.3 <$Revision: 1757674 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient) Completed 100 requests Completed 200 requests Completed 300 requests Completed 400 requests Completed 500 requests Completed 600 requests Completed 700 requests Completed 800 requests Completed 900 requests Completed 1000 requests Finished 1000 requests

Server Software: TreeFrog Server Hostname: 127.0.0.1 Server Port: 8800

Document Path: /mainpage Document Length: 3254 bytes

Concurrency Level: 100 Time taken for tests: 5.844 seconds Complete requests: 1000 Failed requests: 0 Total transferred: 3616000 bytes HTML transferred: 3254000 bytes Requests per second: 171.12 [#/sec] (mean) Time per request: 584.376 [ms] (mean) Time per request: 5.844 [ms] (mean, across all concurrent requests) Transfer rate: 604.28 [Kbytes/sec] received

Connection Times (ms) min mean[+/-sd] median max Connect: 0 5 48.6 0 516 Processing: 31 573 250.1 609 1172 Waiting: 16 481 196.7 547 797 Total: 31 578 252.2 609 1172

Percentage of the requests served within a certain time (ms) 50% 609 66% 625 75% 656 80% 688 90% 750 95% 1094 98% 1141 99% 1156 100% 1172 (longest request)