bheisler / criterion.rs

Statistics-driven benchmarking library for Rust
Apache License 2.0
4.28k stars 290 forks source link

"cargo bench -- <filter>" does not actually filter the benchmarks #763

Open micheleAndreata opened 4 months ago

micheleAndreata commented 4 months ago

Lets suppose I have a function that performs benchmarks on a grid of parameters, hence it requires a lot of time to complete. I also have a function that performs a very simple benchmark. The two functions have ID "slow" and "fast" respectively and the code in my_benchmark.rs is the following:

criterion_group!(benches, bench_slow, bench_fast);
criterion_main!(benches);

Now, if i run the benchmarks with a filter, like cargo bench -- fast, it gets stuck for a while at the beginning and eventually performs the fast benchmark. This is even more evident when i try to revert the order of the bench functions, writing criterion_group!(benches, bench_fast, bench_slow); instead, which executes the fast benchmark immediately and then gets stuck. It seems that it doesn't actually filter the benchmarks but it silently executes them.

Am I doing this wrong? Is this the intended behaviour? Thanks in advance for the help.