codesenberg / bombardier

Fast cross-platform HTTP benchmarking tool written in Go
MIT License
5.91k stars 313 forks source link

'http2: no cached connection was available' when receiving goAway frame during load test #82

Open jhaeyaert opened 2 years ago

jhaeyaert commented 2 years ago

What version of bombardier are you using?

./bombardier --version
bombardier version v1.2.5 darwin/amd64

What operating system and processor architecture are you using (if relevant)?

darwin/amd64

What did you do?

I use bombardier to make some load tests of a remote service and verify that the service meets the performance expectations. The service is exposed in http2 (can fallback to http1 thanks to alpn) and the bombardier load test is run as follow:

./bombardier -c 20 -d 30s --http2 -k -l -m GET https://my-remote-http2-service.com

Everything is working well until I try to check the behavior of my service during a rolling update (check that there is no loss, errors , latency or throughput degradation).

During a rolling update, each pod is deleted 1 by 1 and active connection is gracefully shutdown by sending a goAway frame. All pending requests are honored. Everything is made to keep the service reachable during the rolling update.

The result of the bombardier command shows the following error:

Statistics        Avg      Stdev        Max
  Reqs/sec    375273.79  118131.95  565257.94
  Latency       46.94us   429.41us   247.28ms
  Latency Distribution
     50%    10.00us
     75%    28.00us
     90%   109.00us
     95%   199.00us
     99%   486.00us
  HTTP codes:
    1xx - 0, 2xx - 2241, 3xx - 0, 4xx - 0, 5xx - 0
    others - 11255262
  Errors:
    Get "https://my-remote-http2-service.com": http2: no cached connection was available - 11255262

What you expected to happen?

I was expecting bombardier to be able to create a new connection like it is able to do with http1.1 when it receives a Connection: close header (the same process with --http1 option works like a charm without any error).

What actually happened?

The goAway frame seems received and well interpreted but bombardier keeps trying to reuse the connection which seems to lead to the http2: no cached connection was available error.

Thanks for you help.