search

DigitalInBlue / Celero

C++ Benchmark Authoring Library/Framework
Other
824 stars 95 forks source link

Fixtures are broken from the initial commit #83

Closed peterazmanov closed 8 years ago

peterazmanov commented 8 years ago

TestFixture class calls setUp once before iterations calls to UserBenchmark and tearDown once after. But state of the benchmark has to be cleared each iteration. As a result experiment sortingRandomInts mentioned in docs sorts random array once and then sorts previously sorted array 9999 times for each of 30 samples...

DigitalInBlue commented 8 years ago

What you have reported is true, however this is not a bug. Setup is called between each group of iterations. That is, it is called once per SAMPLE, not per ITERATION. This must be done in order to not measure set up costs. As per the example noted, if you wanted to have the vector in unsorted state for each measurement, then you should move the setup code into the experiment and the experiment's baseline so that it is measured uniformly across all experiments and will cancel itself out in the measurements.

peterazmanov commented 8 years ago

Ok, aforementioned behavior is there by design. In that case ExperimentSortingRandomInts is misleading to say the least. What are we measuring in it? I'm talking about experiment code excerpts from which are on the main page of the project.

DigitalInBlue commented 8 years ago

You are correct. The example needs to be updated. I clearly overlooked that when I developed the example and documentation.