I've noticed - without taking a look at the code - that the benchmarks round numbers too much, or have a lower precision than i expect.
For example if a threshold tolerance is set to 3%, and the result is 2.8% above the thresholds, the benchmark will fail because it'll round 2.8 to 3%, then fail the benchmark since 3% == 3%.
Or for the faster of the benchmarks, I'd like to have microsecond precision instead of millisecond, so i don't need to use a 6% threshold tolerance just because the benchmark could move a bit and round it up to the next 10-millisecond.
I've noticed - without taking a look at the code - that the benchmarks round numbers too much, or have a lower precision than i expect.
For example if a threshold tolerance is set to 3%, and the result is 2.8% above the thresholds, the benchmark will fail because it'll round 2.8 to 3%, then fail the benchmark since 3% == 3%.
Or for the faster of the benchmarks, I'd like to have microsecond precision instead of millisecond, so i don't need to use a 6% threshold tolerance just because the benchmark could move a bit and round it up to the next 10-millisecond.