search

the-benchmarker / web-frameworks

Which is the fastest web framework?
MIT License
6.99k stars 666 forks source link

Choose a sieger #670

Closed waghanza closed 5 years ago

waghanza commented 5 years ago

Hi @fafhrd91,

You said to me (on gitter) that wrk compute latency at max throughput, and it could be more realistic to compute it at constant throughput. Having this information, it could be useful to change (or not) of sieger.

There some :

The question here, is which of these tool (or another I don't know) has the more realistic results (for this use case ?)

Regards,

tsenart commented 5 years ago

Observing latency at different request rates (throughput), as opposed to benchmark the maximum capacity of a system, is what Vegeta was built for. Take it for a spin!

waghanza commented 5 years ago

@OvermindDL1 any suggestion ?

OvermindDL1 commented 5 years ago

Time to benchmark the benchmarkers? ^.^;

waghanza commented 5 years ago

yeah, and it seem to clarify results :stuck_out_tongue:

waghanza commented 5 years ago

This link seems to be interesting => https://github.com/denji/awesome-http-benchmark

waghanza commented 5 years ago

I think about using http//k6.io

@OvermindDL1 any advice ?

OvermindDL1 commented 5 years ago

I tested k6 in #802 and in short, it's not good. ^.^; Overall with the tests of a variety of tools in #802 wrk was still the best overall with apachebench close behind on low thread-counts (though if on many-core CPU's then it started doing pretty badly since it's only single threaded).

OvermindDL1 commented 5 years ago

For note:

You said to me (on gitter) that wrk compute latency at max throughput, and it could be more realistic to compute it at constant throughput. Having this information, it could be useful to change (or not) of sieger.

Testing latency at a couple of constant throughputs is useful but also testing the maximum throughput of a server significantly matters to see how well it handles the primary failure condition and how well it is able to actually pump through requests as fast as possible. A ruby server may be able to answer a request in 20 milliseconds if it's only getting one at a time for example, but if it goes up to 20 seconds to answer a request when under even just a light load is not a great indicator for its reliability...

waghanza commented 5 years ago

Thanks for all comments here, we will use wrk then