search

codesenberg / bombardier

Fast cross-platform HTTP benchmarking tool written in Go
MIT License
5.91k stars 313 forks source link

For some servers wall time significantly exceeds the expected duration of the test when fasthttp client is used #103

Closed cyrusmsk closed 11 months ago

cyrusmsk commented 1 year ago

What version of bombardier are you using?

bombardier version v1.2.6 darwin/arm64

What operating system and processor architecture are you using (if relevant)?

darwin/arm64

What did you do?

./bombardier-darwin-arm64 -c 64 -d 10s -l http://localhost:3000

What you expected to happen?

Max Latency < 10s

What actually happened?

In some runs of different servers I have Max Latency much more than 10s:

Bombarding http://localhost:3000 for 10s using 64 connection(s)
Done!
Statistics        Avg      Stdev        Max
Reqs/sec     17457.81    2832.99   22854.05
Latency        4.31ms   168.35ms     31.41s
Latency Distribution
   50%     2.45ms
   75%     3.06ms
   90%     3.38ms
   95%     3.64ms
   99%     4.47ms

How this could happen? Is there any way to see which request got the Max Latency?

codesenberg commented 1 year ago

Hard to tell really.

http://localhost:3000/

Since it would seem that you know what kind of server makes it possible to get such results, care to elaborate on this a bit?

cyrusmsk commented 1 year ago

Yes, sure. I will double test results with native Darwin and Linux(Docker) until the weekend. The source code and previous results are available here: https://github.com/cyrusmsk/d_http_benchmark/tree/main

The only thing that currently build instruction optimized for Docker run. To run locally on macOS it's better to remove "flags-ldc" row from dub.json file.

codesenberg commented 1 year ago

Thanks for the info, @cyrusmsk.

I did some troubleshooting and my current understanding is that this has more to do with the HTTP client used, rather than with the server used. With --fasthttp (default client) the test indeed can take unexpectedly long to complete, but with --http1 & --http2 the test usually completes within ~--duration + --timeout + time to prepare the test + time to shutdown the test (last 2 usually don't take long, i.e. couple seconds at most). Not sure what causes shutdown delay for fasthttp. Could be that some timeouts/deadlines we set are not respected by the client or that bombardier is not setting up the client so that they are respected. Will have to dig deeper.

The issue itself seems to be more of an annoyance and thankfully there is a workaround.