google / benchmark

A microbenchmark support library
Apache License 2.0
8.6k stars 1.57k forks source link

[BUG] PauseTiming does not work #1674

Closed plzfgme closed 9 months ago

plzfgme commented 9 months ago

Describe the bug

I have written a minimal piece of code to reflect the problem I am experiencing.

According to my expectations, the following two benchmarks should give similar results, since I used PauseTiming before doing heavy work. However, Bench1's result is much lower than Bench2's and I don't know why.

static void Bench1(benchmark::State &state) {
  for (auto _ : state) {
    state.PauseTiming();
    for (std::size_t i{0}; i < 10000000; i++) {
      std::size_t j{0};
      j++;
      benchmark::DoNotOptimize(j);
    }

    state.ResumeTiming();
    for (std::size_t i{0}; i < 10; i++) {
      std::size_t j{0};
      j++;
      benchmark::DoNotOptimize(j);
    }
    state.PauseTiming();
  }
}
BENCHMARK(Bench1)->Unit(benchmark::kMillisecond);  // NOLINT

static void Bench2(benchmark::State &state) {
  for (auto _ : state) {
    state.PauseTiming();
    for (std::size_t i{0}; i < 10000000; i++) {
      std::size_t j{0};
      j++;
      benchmark::DoNotOptimize(j);
    }

    state.ResumeTiming();
    for (std::size_t i{0}; i < 10; i++) {
      std::size_t j{0};
      j++;
      benchmark::DoNotOptimize(j);
    }
    state.PauseTiming();

    for (std::size_t i{0}; i < 10000000; i++) {
      std::size_t j{0};
      j++;
      benchmark::DoNotOptimize(j);
    }
  }
}
BENCHMARK(Bench2)->Unit(benchmark::kMillisecond);  // NOLINT

Benchmark result: https://quick-bench.com/q/LkIooVDDhuq7CQ5nSkvDtYclybc

plzfgme commented 9 months ago

It seems that PauseTiming must be followed by a ResumeTiming.