Design a good benchmark

[Geal]

In gitlab by @Geal on Mar 9, 2016, 18:02

Current tests are done on a 512MB droplet from Digital Ocean, with HTTP requests generated by ab or wrk, from a Dedibox. The proxy is forwarding requests to the Clever Cloud frontend, with the hostname "http://wp-perf-test.cleverapps.io" or "http://rust.cleverapps.io".

Issues with this:

• the wp-perf-test website is a Wordpress, so it has a very high latency by default
• the rust website returns a very small response, it does not look like real traffic
• ab is too limited
• wrk does not give much indication of why some requests failed
• the digital ocean droplet is too small

Ideally, a benchmark would compare to haproxy, with load balancing on multiple backends, and it would gradually increase the number of concurrent requests, until the proxy starts failing. That benchmark would also record perf info, with sysdig or something else.

[Geal]

In gitlab by @divarvel on Mar 9, 2016, 18:05

The proxy server should be near clever cloud to be closer to actual use cases

We could add other websites running on clever cloud (eg play websites we can duplicate)

Gatling could be a better tool for stress testing

[Geal]

In gitlab by @Geal on Nov 5, 2016, 15:22

draft for a testing protocol:

• a few machines running nginx to serve a static website, with various file sizes (between 100kB and 5MB should be good)
• another machine, very close, with multiple cores available, for the proxy (preferably not a virtual machine)
• a few machines to hold the clients
• try to put all the machines as close to each other as possible

The tests should be performed in 3 setups:

• clients talking directly to the nginx backends (use DNS round robin to reach all of them?)
• testing with our proxy, haproxy and traefik

Trying with a direct connection to the backend is useful to know how much latency is introduced by the proxies.

The clients could use wrk2 for performance testing, since the percentile calculation looks correct. httperf and autobench can be useful to get information on the number of errors

Parameters to modify during the tests:

• number of cores used. single core testing is useful to get deterministic tests and results, and calculate capacity
• max number of sessions in proxy config (to test the behaviour when we reach capacity)
• keep alive or not (httperf lets you specify how many requests will be performed per connection)
• number of concurrent connections
• number of requests per second

For a specific number of connections, do a serie of tests with increasing requests per second. Do not stop until performance is degrading, connections are refused or errors appear (and keep pushing further).

How to graphically represent such tests? For one number of concurrent connections, draw curves of the latency percentiles depending on the number of requests per second? on a graph detailing RPS depending on nb of concurrent connections, show the quantity of errors or lost connections with a color gradient?

Which kernel settings must be tuned to get good performance?

[Geal]

In gitlab by @Geal on Nov 30, 2016, 11:33

resources: https://www.urbanairship.com/blog/linux-kernel-tuning-for-c500k https://www.urbanairship.com/blog/c500k-in-action-at-urban-airship https://mrotaru.wordpress.com/2013/10/10/scaling-to-12-million-concurrent-connections-how-migratorydata-did-it/

[NotBad4U]

I started to make the benchmark infrastructure based on docker with docker-compose. The purpose is to make the benchmarks easy to reproduce for any one.

Link to the repository: proxy-benchmarks

Overview

                                                           +-----------------+
                                                           |                 |
                                                           | nginx backend   |
                                                     +---> |                 |
                                                     |     +-----------------+
+------------------+         +-------------------+   |
|                  |         |                   |   |
| wrk2 client      +---------| Proxy             |   |     +-----------------+
|                  |         |                   |   |     |                 |
|                  |         |                   +---+---->| nginx backend 2 |
+------------------+         |                   |   |     |                 |
                             +-------------------+   |     +-----------------+
                                                     |
                                                     |      +-----------------+
                                                     |      |                 |
                                                     +----->| nginx <n>       |
                                                            |                 |
                                                            |                 |
                                                            +-----------------+

nginx (backend)

They serve static json content. They have different endpoints where the response is a json with various file sizes (between 100kB and 5MB).

Optimisation made for sending file:

• sendfile (The most important)
• tcp_nopush
• tcp_nodelay
• A short keepalive_timeout

Proxy

The proxy supported for this benchmarks will be:

• Sozu
• Haproxy
• Envoy
• Traefik

NOTE: We can add more in the future.

wrk2 client

The client will send a large amount of request on the differents nginx endpoints (The response size will change). We'll use the LUA Jit to change the endpoint to request dynamically.

Examples:

• wrk-http-benchmarking-tool-example
• multiplepaths.lua

NOTE: We should take a look at siege ( an http load tester and benchmarking utility).

Launch the benchmarks

TODO

Benchmarks configurations

TODO

Output format result

TODO

[Geal]

here's the kind of measurements I'd like to see:

• fix a specific size for the proxy (like, one core, X GB of RAM)
• fix a maximum amount of concurrent requests and requests per second
• start enough backend servers to handle that maximum traffic (to make sure the benchmark won't be limited by the backends, because a proxy does less work than a web server)
• vary one of the variables, like the number of requests per second, starting with a low number, until the previously defined maximum
• on each value, calculate the latency histogram and store it
• ideally, we increase the value until the proxy latency starts to increase (to see the point of failure)
• graph the latencies depending on the reqs/s: one line per latency value (p50, p90, etc)

The benchmarks should be easy to reproduce (like, "use this docker-compose file to reproduce"), and allow comparison between different runs (to see if sozu improves, or compare to other proxies)

[Geal]

See https://github.com/kellabyte/go-benchmarks/issues/6#issuecomment-386714524 https://github.com/kellabyte/go-benchmarks/blob/master/queues/README.md and https://twitter.com/gcouprie/status/992522877835579397?s=19 for graph examples

[WhyNotHugo]

There's a really good benchmark of nginx vs caddy here:

https://blog.tjll.net/reverse-proxy-hot-dog-eating-contest-caddy-vs-nginx/

Plus the setup is all made public. Maybe piggybacking on this would yield some useful results?