umakanta75 / caliper

Automatically exported from code.google.com/p/caliper
Apache License 2.0
0 stars 0 forks source link

Provide a better solution for non deterministic allocation behavior #273

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago
Currently caliper will aggressively check for non deterministic allocation 
behavior and fail the measurement when non determinism is detected.

This can be partially worked around by setting 
instrument.allocation.options.trackAllocations=false
which simply reduces the amount of data that caliper tracks and thus limits the 
kinds of non determinism that can be detected.

On the one hand a non deterministic benchmark is a bad thing since it makes 
your measurements suspect so users should try to eliminate it (i.e. make the 
benchmark perform the exact same amount of work on each rep).  However, not all 
non-determinism (of allocation behavior) can be eliminated.  For example:

In multithreaded benchmarks, lock contention will cause allocations and is 
inherently non deterministic.
Lazy initialization of data is a common pattern and its possible that not all 
such initialization can be triggered in warmup.
and others

Caliper should try to support such use cases.

One proposal would be for the AllocationWorker to be configured to work like 
the RuntimeWorker and instead of subtracting a baseline measurement to remove 
overheads just run a large number of reps to minimize them.  This could be 
configurable via something like
instrument.allocation.options.strategy

Original issue reported on code.google.com by lu...@google.com on 16 Aug 2013 at 4:22