cartographer-project / point_cloud_viewer

View billions of points in your browser.
Apache License 2.0
339 stars 98 forks source link

Use criterion for benchmarks #429

Closed nnmm closed 4 years ago

nnmm commented 4 years ago

With bencher, it was hard to get meaningful results:

Criterion can be used like bencher (cargo bench) but has much nicer reporting. Sample output:

Benchmarking bench_s2_building_singlethreaded: Warming up for 3.0000 s Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 828.6s or reduce sample count to 10. bench_s2_building_singlethreaded time: [148.67 ms 149.33 ms 150.27 ms] change: [-6.5982% -3.3366% +0.3522%] (p = 0.06 > 0.05) No change in performance detected. Found 16 outliers among 100 measurements (16.00%) 14 (14.00%) high mild 2 (2.00%) high severe

As you can see, if you've run it before, it will compare the durations. It also does warmup runs, which helps decrease variance from disk caches.

nnmm commented 4 years ago

ptal @feuerste

feuerste commented 4 years ago

... and of course formatting needs to be fixed first :)

nnmm commented 4 years ago

True, it's heavy, but at least it's a dev-dependency. So if you do cargo build in any of the crates including point_cloud_test, it won't get built, only when doing cargo test/bench in that crate (or all crates via --workspace). Do you agree it's worth it?

feuerste commented 4 years ago

It's fine with me.