AndyAyersMS
commented
9 years ago If you are going to start in on statistics to deal with noisy metrics, I would recommend looking at something like a bootstrap to estimate metrics and their confidence intervals from limited sized samples. For instance you could increase iterations until the confidence interval of your key metric (median time, say) from repeated runs at that number of iterations is small enough for your liking. And if this doesn't work out, you'd know you have a noisy test with some uncontrolled influence.
Bootstrap can also be used to compute p-scores when comparing two sets of runs to see if there is a statistically significant regression or improvement in the metric. I have some sample code I could donate if you're interested.
There has also been some good work on randomization to support statistically sound performance measurements, see for instance Curtsigner and Berger's STABILIZER, but this may be a bit more to bite off than you all had in mind.
pharring
ericeil
sajayantony
dmetzgar
Several other issues have mentioned how iterations should be handled, so I thought this warranted a dedicated issue.
The basic question is, for any given test, how many times should it be run? My thinking on this is as follows:
It should be trivial to author a simple perf test that yields useful results.
I should be able to write something like:
...and get a useful measure of that method's performance.
A useful result typically requires multiple iterations.
Performance, particularly in .NET, is inherently non-deterministic. The only way to get a useful idea of the performance of a given piece of code is to run it multiple times, and then use statistical techniques to characterize the data gathered.
It's not generally possible to predict the number of iterations required, ahead of time.
The statistical methods used determine how many "sample" measurements are needed, and this will vary depending on several factors:
a) The particular mix of operations in the test itself. Some methods are simple, single-threaded, CPU-bound, and therefore are not inherently non-deterministic, but most will end up allocating (involving the GC, which is non-deterministic), interacting with other threads, interacting with I/O devices, etc.
b) The state of the machine on which the test is running. It's nearly impossible to isolate the execution of a single process from everything else happening on/to the machine. For example, a sudden surge in network traffic may consume CPU resources for interrupt processing, making the test code seem to take longer for the duration of the surge. Or, there may be more physical memory usage in other processes on one machine vs. another, or at one time vs. another, causing the GC to collect more aggressively, temporarily.
The effects in (a) are largely predictable, though it may require a fair amount of experimentation to find an iteration count that consistently leads to useful results. However, the effects in (b) are completely unpredictable, especially in a system that needs to work reliably outside of the controlled environment of a dedicated performance lab.
Performance testing should be fast
On the CLR team, we've traditionally compensated for the unpredictability in perf tests by hard-wiring them for a large number of iterations. Much of the time this ends up being a waste of (computer) time. If we have to run every test enough times that we could get a useful result if we were running in a noisy environment, we limit the number of tests we can run in a clean environment, in a given amount of time. Also, we've ended up running many relatively stable tests as if they were unstable, simply because it's a pain (and time-consuming) to go analyze every test case and determine the optimal number of iterations; so we run even the stable, predictable tests more than needed, further wasting machine time on every run.
The wasted machine time becomes a much larger issue when we start asking OSS contributors to run these tests on their own machines prior to submitting changes.
The above, to me, argues for a dynamic system that compensates for unpredictability at run-time, by default. We should be running the test until we have some confidence that we have a useful result, for that particular run. I think the above also argues that it's harmful to offer any kind of mechanism for hard-wiring particular iteration counts, as they will inevitably lead to tests that either don't produce useful results, or waste time producing overly-accurate results. I can see configuration of extreme upper limits, as "back-stops" against the run-time algorithms going astray, but I worry that those would be quickly, and quite accidentally, abused. A backstop might be better implemented as an overall execution time limit; if the test run exceeds that limit, it simply fails, rather than producing useless, but seemingly valid, results.
Of course, that's just one point of view. I'm eager to hear others.