Open kevincox opened 11 months ago
I have the same problem, except it does not complete in reasonable time, the code:
use criterion::{black_box, criterion_group, criterion_main, BenchmarkId, Criterion};
use rand::prelude::*;
pub fn lookup(c: &mut Criterion) {
let mut group = c.benchmark_group("lookup");
for size in (0..4*1024*1024).step_by(1024*1024) {
let mut rng = thread_rng();
let mut data: Vec<u64> = vec![0; size];
rng.fill(&mut data[..]);
let mut keys = data.clone();
keys.shuffle(&mut rng);
group.bench_with_input(BenchmarkId::new("linear", size), &size, |b, size| {
b.iter(|| {
for key in &keys {
black_box(data.contains(key));
}
});
});
let mut rng = thread_rng();
let mut data: Vec<u64> = vec![0; size];
rng.fill(&mut data[..]);
let mut keys = data.clone();
keys.shuffle(&mut rng);
group.bench_with_input(BenchmarkId::new("constant", size), &size, |b, size| {
b.iter(|| {
for key in &keys {
black_box(data.contains(key));
}
});
});
}
group.finish();
}
criterion_group!(benches, lookup);
criterion_main!(benches);
I tried to reduce number of samples and confidence level like this: group.confidence_level(0.6).sample_size(12);
but it didn't really help.
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 15150.2s, or reduce sample count to 10.
Benchmarking lookup/linear/1048576: Collecting 100 samples in estimated 15150 s (100 iterations)
There is a chance that I am using it wrong but after reading the manual and API docs I can't find a good option to get rid of the warning.
The problem is that I have slow things that I want to benchmark. The tests are on the order of 10s. I could potentially use smaller examples (and I do have some smaller ones in other benchmarks) but I am quite concerned with how long the complex problems take so find it important to have benchmarks for these. Really what I want to do is run 5-10 samples and then use criterion's comparison features to identify changes. Right now my benchmark looks like this:
It seems that the suggestion wants me to tell criterion how long I think my benchmark will take. But I really don't want to bother keeping a runtime estimate up to date. Even worse since I have multiple inputs the duration can be significantly different for each. It seems that I either set it too low and get an annoying warning in my output or I set it too high and further slow down an already slow benchmark.
I understand that with a small number of samples the confidence won't be particularly high but it seems like the best option available without having the benchmark run for an hour. So I just want to turn off the warning to make reading the output easier.