kornelski
commented
1 month ago PLEASE offer an alternative to the command-line arguments. They are fundamentally broken and unfixable.
For example, https://github.com/gfx-rs/wgpu/blob/trunk/benches/Cargo.toml does everything right, and still doesn't work:
error: unexpected argument found
error: bench failed, to rerun pass `--bench root`
I need to git bisect a performance regression, and too late after the fact I've realized that the default of benchmark with 100 iterations takes way too long to finish the bisect in a reasonable timeframe. The difference I'm measuring will be measurable with just a handful of iterations, so I would be beneficial if I could lower it for the bisect operation. The problem is that change of Criterion's configuration requires significant source code changes. This is especially inconvenient when measuring past commits that can't be simply modified without some git wrangling.
My second use-case is setting a baseline without running into #193. In bigger workspaces it's hard to track down all the places that break the custom command-line arguments. Setting extra criterion options via env vars would easily bypass the picky test runner.