vegeta cannot get the best out of the APIs

martin-g / http2-server-perf-tests

Test the performance of different HTTP2 servers

MIT License

2 stars 3 forks source link

vegeta cannot get the best out of the APIs #1

Closed omid closed 4 years ago

omid commented 4 years ago

AFAICS, vegeta cannot get the best out of the APIs.

I've tried wrk with the following command, I've got ~4 times more requests for the go code, and ~8 times more requests for the rust code within 20 seconds. 99 percentile for rust is 15.5ms, for go is 7.5ms. But go returns some errors. At 75 percentile, rust is about 2 times better than go.

wrk -t12 -c100 -d20s --latency https://localhost:18080

Sure you can change and tweak the command based on your need!

martin-g commented 4 years ago

Hi,

I've explained in the blog article why I don't use wrk:

As a load test client I use Vegeta because wrk does not support HTTP2 and because it has a bug in the way it calculates the statistics.

The bug is explained here: https://medium.com/@martin.grigorov/compare-varnish-cache-performance-on-x86-64-and-aarch64-cpu-architectures-cef5ad5fee5f

So, you tested HTTP1.1, not HTTP2.

Same test executed with Vegeta and HTTP1.1 gives 35000 reqs/sec for Tomcat NIO, i.e. twice better.

omid commented 4 years ago

Then maybe find a better client that can handle this much requests. All ratings are so similar, seems like a client limitation.

I've tried h2load, here is the result on my local machine h2load -t5 -c100 -D20 https://localhost:18080:

Rust: ~5 million reqs (~5 times more than vegeta) minimum (runtime) memory usage: 1MB max memory usage: 5MB
Go: ~3 million reqs (~3 times more than vegeta) minimum (runtime) memory usage: 12MB max memory usage: 30MB

martin-g commented 4 years ago

If I use h2load then I won't be able to compare the results against HTTP 1.1 :-/ The load client behaves the same way for all tested servers, so it gives you relative comparison.

All I need is a tool that supports all protocols (HTTP 1.1, HTTP2 and h2c). Vegeta does that! Vegeta also gives you statistics - latency (with percentiles), throughput, rate, errors, bytes, ...

So far I haven't found a better (and free) HTTP load testing tool.

omid commented 4 years ago

And Vegeta cannot put the pressure on the APIs. So there is no tool that can work :) And the result is invalid.