Open GoogleCodeExporter opened 9 years ago
Some thoughts
1. Loaded machine = out of scope. Assume tests are running on dedicated
hardware, and if not then our threshold offering will be good enough.
2. Running on different machines from build to build. Do not allow it! We could
enforce this by failing build if environment properties change, or we could
ignore it and assume the environment is the same. Latter is more flexible and
more risky.
3. Thresholds. My favorite idea today is letting user specify a rule as a
Groovy script on the job config page which returns true or false:
"val < prev" (better than prev run, aggressive!)
or "val <= max(10)" (not worse than worst of last 10 runs)
or "val <= min(5) * 1.05" (within 5% of best of last 5 runs)
Next to the input box on the job config page I'd show (or link to) the set of
predefined variables/functions.
I've embedded a Groovy console in Swing project before (scroll down @
http://code.google.com/p/bond-valuation/) and it worked well and was simple.
Jenkins already has a Groovy dependency in some form, I think.
Original comment by brianfromoregon
on 21 Nov 2011 at 8:35
About point #2 above. Another solution is Caliper addressing this:
http://code.google.com/p/caliper/issues/detail?id=138
Original comment by brianfromoregon
on 21 Nov 2011 at 8:37
Step 1 is done, timing result trends are being displayed. Step 2 is actually
failing the build with user defined rules (groovy).
Original comment by brianfromoregon
on 19 Jan 2012 at 11:57
Original comment by brianfromoregon
on 8 Feb 2012 at 2:18
I'm aware of a use case for which the mean measurement is insufficient for
doing regression analysis. This use case wants to do regresssion analysis on
the 99 percentile value in addition to mean.
Original comment by brianfromoregon
on 10 May 2012 at 6:09
Original issue reported on code.google.com by
brianfromoregon
on 28 Oct 2011 at 5:28