Parameters in `/etc/sysctl.conf`

[hpoettker]

Hi @ebarlas, thanks for this great comparison!

Did you have to tweak parameters in /etc/sysctl.conf on the EC2 instances to get the concurrency level of 20,000 to work? If yes, could you add the parameters to the repository? That would be very interesting to see.

[ebarlas]

Thanks for raising this. I'm surprised I omitted that detail. I'll review my notes and update the docs in a day or two.

[ebarlas]

I reviewed my notes and didn't find any mention of kernel options for this particular experiment. I had to double check because other recent experiments, such as project-loom-c5m, relied heavily on kernel options.

Just to be certain, I fired up three new EC2 instances and ran several of the benchmarks. The results are remarkably consistent with the original findings.

ab output from concurrency 20,000 with threads:

$ ab -k -c 20000 -n 2400000 -H "Authorization: token" http://10.39.197.146:9000/meetings
This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 10.39.197.146 (be patient)
Completed 240000 requests
Completed 480000 requests
Completed 720000 requests
Completed 960000 requests
Completed 1200000 requests
Completed 1440000 requests
Completed 1680000 requests
Completed 1920000 requests
Completed 2160000 requests
Completed 2400000 requests
Finished 2400000 requests

Server Software:        
Server Hostname:        10.39.197.146
Server Port:            9000

Document Path:          /meetings
Document Length:        51 bytes

Concurrency Level:      20000
Time taken for tests:   476.727 seconds
Complete requests:      2400000
Failed requests:        0
Keep-Alive requests:    2400000
Total transferred:      350400000 bytes
HTML transferred:       122400000 bytes
Requests per second:    5034.32 [#/sec] (mean)
Time per request:       3972.729 [ms] (mean)
Time per request:       0.199 [ms] (mean, across all concurrent requests)
Transfer rate:          717.78 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   42 485.6      0    5988
Processing:  1277 3879 1258.1   3730   16255
Waiting:     1241 3879 1258.1   3730   16255
Total:       1293 3922 1394.3   3738   16894

Percentage of the requests served within a certain time (ms)
  50%   3738
  66%   4370
  75%   4505
  80%   4755
  90%   5408
  95%   5924
  98%   6957
  99%   9635
 100%  16894 (longest request)

ab output from concurrency 20,000 with virtual threads:

$ ab -k -c 20000 -n 2400000 -H "Authorization: token" http://10.39.197.146:9000/meetings
This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 10.39.197.146 (be patient)
Completed 240000 requests
Completed 480000 requests
Completed 720000 requests
Completed 960000 requests
Completed 1200000 requests
Completed 1440000 requests
Completed 1680000 requests
Completed 1920000 requests
Completed 2160000 requests
Completed 2400000 requests
Finished 2400000 requests

Server Software:        
Server Hostname:        10.39.197.146
Server Port:            9000

Document Path:          /meetings
Document Length:        51 bytes

Concurrency Level:      20000
Time taken for tests:   211.273 seconds
Complete requests:      2400000
Failed requests:        0
Keep-Alive requests:    2400000
Total transferred:      350400000 bytes
HTML transferred:       122400000 bytes
Requests per second:    11359.72 [#/sec] (mean)
Time per request:       1760.607 [ms] (mean)
Time per request:       0.088 [ms] (mean, across all concurrent requests)
Transfer rate:          1619.65 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   41 474.3      0    5859
Processing:  1100 1680 498.9   1591    7380
Waiting:     1100 1680 498.9   1591    7380
Total:       1100 1721 915.1   1591   12563

Percentage of the requests served within a certain time (ms)
  50%   1591
  66%   1662
  75%   1723
  80%   1768
  90%   1873
  95%   1977
  98%   2293
  99%   5040
 100%  12563 (longest request)

[hpoettker]

Thanks for having a look! And also for posting the percentiles. It's really appreciated.

I've been re-implementing the comparison in a bit of an enterprise edition. I had stability problems with the servlet apps that I tackled with little success by tweaking parameters. But after reading your reply, I concluded that c5.2xlarge is simply too small for the servlet apps under high load. On a c5.4xlarge instance for the frontend, everything runs smoothly without any tweaking.