Document how to deal with "ERROR: Hotspot compilation occurred during timing. Warmup is likely insufficent." when running tests

[GoogleCodeExporter]

What steps will reproduce the problem?
1. I run the code below with no additional arguments
2.
3.

What is the expected output? What do you see instead?
Caliper completes without printing to stderr.
Instead, during one of the experiments, Calper prints: 

"ERROR: Hotspot compilation occurred during timing. Warmup is likely 
insufficent."

What version of the product are you using? On what operating system?
a3703ae4544b3952e0944fd8d7e3dcd16f72c3fc

java version "1.7.0_17"
Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)

Please provide any additional information below.

code:

import com.google.caliper.Benchmark;
import com.google.caliper.runner.CaliperMain;

public class TestFinally extends Benchmark {
    public long timeNanoTime(int reps) {
        long ret = 0;
        for (int i = 0; i < reps; i++) {
            ret += System.nanoTime();
        }
        return ret;
    }

    public long timeCurrentTimeMillis(int reps) {
        long ret = 0;
        for (int i = 0; i < reps; i++) {
            ret += System.currentTimeMillis();
        }
        return ret;
    }

    public static void main(String[] args) throws Exception {
        CaliperMain.main(TestFinally.class, args);
    }
}

Original issue reported on code.google.com by agrothberg on 8 Apr 2013 at 12:03

[GoogleCodeExporter]

Sorry for letting this one slip through the cracks.  The issue is that some 
method is getting JIT'd in the middle of the timing process.  I'm writing up 
the docs on how to deal with this type of thing, but here are the high-level 
steps for dealing with this.

1) run with --verbose to see _what_ is getting JIT'd.  Given your code, I'd 
guess that it's probably something random and has nothing to do with your 
benchmark. :-(

2) Random or not, the way to deal with this is to adjust the timing interval 
and the warmup.  If something called by your benchmark code is being JIT'd, you 
need more warmup.  This is done by setting the 
"instrument.micro.options.warmup" configuration property.  It takes a value 
like "5s" or "100ms".  Either set the value in ~/.caliper/config.properties or 
using the -C flag.  If it's a random method, I've found that decreasing the 
timing interval (instrument.micro.options.timingInterval, same value format) 
and the warmup is usually the way to go.  Essentially, the JVM is running for 
_so_ long that thing that get invoked very, very infrequently happen to get 
JIT'd, so you just make the JVM run for less time total.  Run your benchmark 
with --print-config to see what the current values are and adjust them from 
there.

HTH.

Original comment by gak@google.com on 17 Apr 2013 at 6:32

• Changed title: Document how to deal with "ERROR: Hotspot compilation occurred during timing. Warmup is likely insufficent." when running tests
• Changed state: Accepted