Currently caliper will aggressively check for non deterministic allocation
behavior and fail the measurement when non determinism is detected.
This can be partially worked around by setting
instrument.allocation.options.trackAllocations=false
which simply reduces the amount of data that caliper tracks and thus limits the
kinds of non determinism that can be detected.
On the one hand a non deterministic benchmark is a bad thing since it makes
your measurements suspect so users should try to eliminate it (i.e. make the
benchmark perform the exact same amount of work on each rep). However, not all
non-determinism (of allocation behavior) can be eliminated. For example:
In multithreaded benchmarks, lock contention will cause allocations and is
inherently non deterministic.
Lazy initialization of data is a common pattern and its possible that not all
such initialization can be triggered in warmup.
and others
Caliper should try to support such use cases.
One proposal would be for the AllocationWorker to be configured to work like
the RuntimeWorker and instead of subtracting a baseline measurement to remove
overheads just run a large number of reps to minimize them. This could be
configurable via something like
instrument.allocation.options.strategy
Original issue reported on code.google.com by lu...@google.com on 16 Aug 2013 at 4:22
Original issue reported on code.google.com by
lu...@google.comon 16 Aug 2013 at 4:22