msmith-techempower
commented
6 years ago I inspected a number of results, looking at onion with regard to latency and rps specifically, and it seems very volatile toward the former, but rather constant to the latter.
To me, this suggests that perhaps there is an issue with wrk and how it captures latency results. This has been brought up before and cited as the impetus for wrk2.
It would require a non-trivial amount of trial and error, not to mention evaluating the results, but we might be positioned well to take another look at wrk2 to see if the latency numbes are (fuzzily defined) 'better' while the requests per second (the main capture point about which we care) remain consistent.
dralley
I'm looking at the visualization of results from continuous tests f26afdb3-7c19-4c20-84de-228b10434f84 and 3da523ee-fff1-45d8-9044-7feb532bf9ee, two recent consecutive runs.
I noticed in the JSON Latency chart that the onion framework went from being second best in latency (0.1ms, 0.0 stdev, 4.2ms max) to being fourth from worst (118.7ms, 644.4ms stdev, 6420.0ms max). That's a difference of 3 orders of magnitude, and to the best of my knowledge there was no change to this framework between these runs.
I'm not familiar with this specific framework and have no idea why there is such a huge difference, but it stood out as something that may be more than just random noise, and possibly an indication of a deeper flaw in the latency benchamrks, continuous or otherwise.
@nbrady-techempower recommended I open an issue here to track the investigation of this issue, which may or may not be significant, so here it is :-)