search

brianfromoregon / caliper-ci

Automatically exported from code.google.com/p/caliper-ci
0 stars 0 forks source link

Do regression analysis on timing measurements #5

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
This means caring about which build agent a test ran on, how loaded the agent 
is, and flexible thresholds.

Complicated feature that I don't know how to tackle.

Original issue reported on code.google.com by brianfromoregon on 28 Oct 2011 at 5:28

GoogleCodeExporter commented 9 years ago 
Some thoughts

1. Loaded machine = out of scope. Assume tests are running on dedicated 
hardware, and if not then our threshold offering will be good enough.

2. Running on different machines from build to build. Do not allow it! We could 
enforce this by failing build if environment properties change, or we could 
ignore it and assume the environment is the same. Latter is more flexible and 
more risky.

3. Thresholds. My favorite idea today is letting user specify a rule as a 
Groovy script on the job config page which returns true or false:
"val < prev" (better than prev run, aggressive!) 
or "val <= max(10)" (not worse than worst of last 10 runs)
or "val <= min(5) * 1.05" (within 5% of best of last 5 runs)

Next to the input box on the job config page I'd show (or link to) the set of 
predefined variables/functions. 

I've embedded a Groovy console in Swing project before (scroll down @ 
http://code.google.com/p/bond-valuation/) and it worked well and was simple. 
Jenkins already has a Groovy dependency in some form, I think.

Original comment by brianfromoregon on 21 Nov 2011 at 8:35

GoogleCodeExporter commented 9 years ago 
About point #2 above. Another solution is Caliper addressing this: 
http://code.google.com/p/caliper/issues/detail?id=138

Original comment by brianfromoregon on 21 Nov 2011 at 8:37

GoogleCodeExporter commented 9 years ago 
Step 1 is done, timing result trends are being displayed. Step 2 is actually 
failing the build with user defined rules (groovy).

Original comment by brianfromoregon on 19 Jan 2012 at 11:57

GoogleCodeExporter commented 9 years ago

Original comment by brianfromoregon on 8 Feb 2012 at 2:18

GoogleCodeExporter commented 9 years ago 
I'm aware of a use case for which the mean measurement is insufficient for 
doing regression analysis. This use case wants to do regresssion analysis on 
the 99 percentile value in addition to mean.

Original comment by brianfromoregon on 10 May 2012 at 6:09