Throughput stats is misleading

[vitaly-castLabs]

What version of bombardier are you using?

v1.2.6 darwin/arm64

What did you do?

I run https://github.com/qicosmos/cinatra/blob/master/example/benchmark.cpp against ./bombardier -c 100 -n 500000 http://localhost:8090 with and without dummy headers:

1. No cheating, original code: throughput is ~40 MBps;
2. Option A. Add a dummy header to the response: throughput is now ~160 MBps:

   resp.add_header("x-perf-dummy", "x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy-x-perf-dummy");

3. Option B. Pad the body (curl http://localhost:8090/ will still return Hello, world! with this) and get throughput jump to ~53 MBps:

   resp.set_status_and_content(status_type::ok, "Hello, world!\rHello, world!\rHello, world!\rHello, world!\rHello, world!");

My point is

Throughput alone is very unreliable - even without explicit cheating a server setting a bunch of optional headers on responses will have a major advantage. What I'd do is:

• Report performance in requests per seconds foremost;
• Calculate throughput only using the body size, headers excluded. Maybe also add a warning if Content-Length is not 13

[sebastienros]

I personally don't take throughput as an indicator of performance, but a way to check that the network is not a bottleneck and that the benchmark works as expected. And also think that headers should definitely be part of this measure otherwise it would have no value.

We agree that RPS is the metric to take into account, not throughput.