Open alexcrichton opened 8 years ago
Nominating for discussion in 1.6, and cc https://internals.rust-lang.org/t/bench-status/2122/10, a thread about this.
I've currently extracted libtest and I would personally like to deprecate #[bench]
with a little bit of extra support from perhaps Cargo or the compiler. We'd basically be saying "let's have benchmark support develop externally first" before moving it back in to the compiler.
I like bench
being as easy to use as it currently is, but would like more functionality.
IMO it would be nice to be able to provide custom providers for --test
bineries so you could compile rustc … --test
and run binary --bench
, binary --quickcheck
, binary --criterion
etc for whatever testing and benching libraries you’re using.
Yeah my feeling here is that I wouldn't want to stabilize anything in the compiler itself unless it looks like a generic "custom test framework" infrastructure, but that's unfortunately a big undertaking that may take awhile. As a result my personal preference is to deprecate the support in the meantime for technically-the-same-if-not-quite-as-easy-to-use support on crates.io
The libs team decided to punt on this for 1.6
This looks like it might be a good replacement for the current #[bench]
mechanism: https://github.com/japaric/criterion.rs.
I agree with Alex that deprecation is the best resolution.
This just came up on IRC and should be a consideration for anything we do here. Allowing for "unbenchmarked" parts of an iteration. The specific case this came up for was benchmarking an in-place sorting algorithm. Since performance is going to be different for sorted vs. unsorted lists, you need to clone the list each iteration which can throw off the results.
I'm sorry, but this is still required in nightly 1.20. If I write:
#[bench]
fn bench_blur_image(b: &mut Bencher) {}
Rust complains that it can't find Bencher
. You have to use #![feature(test)]
and extern crate test
still. Why is this not stabilized or that Bencher
could be found automatically? Just asking.
@fschutt That is correct, benchmarking with #[bench]
is unstable on only available in Nightly at the moment. This issue is about tracking its stabilization.
What's the current status of this given the recent discussion here? Is there another ongoing discussion somewhere else?
Latest development seems to be https://github.com/rust-lang/rfcs/pull/2318
Triage: no major movement recently
This has been discussed numerous times across various RFCs and internals threads over the last ~7 years (and probably before then, too), and the latest 'next step' towards stabilization here seems to be the custom test frameworks RFC (tracked in https://github.com/rust-lang/rust/issues/50297). That RFC and implementation seem unlikely to be pushed over the finish line -- even for a working, usable nightly implementation -- on any relatively short timescale, so I would like to suggest that we discuss whether the existing bench API could be minimally stabilized essentially as-is (listed in the issue description).
It has a number of extensions that users have asked for historically, some already implemented:
The majority of these seem relatively feasible to either add to libtest or develop out of tree (e.g., in crates like criterion), but also look to me like extensible design atop the very simple interface we have today which has -- largely -- worked fine for the majority of basic use cases. A relatively fast skim through of the various RFCs to me suggest that the main objection to just stabilizing in place has always been more general designs or concerns about a forever stable piece of the language. Ultimately, it seems to me that a few functions and an attribute (which could have a different meaning or multiple meanings in the future) are not that painful to commit to.
For example, we could, I think, easily imagine that #[bench]
here is basically the equivalent of #[ignore]
. The main difference is the argument passed to the function, but it seems like the minimal API being stabilized here could be provided by other frameworks, or at least doesn't really hurt them. The surface area is pretty minimal, after all.
The stabilization would mean stable access to the test crate, which I believe today is entirely unstable, so one alternative would be to move Bencher to std::test, re-exporting it in test.
@Mark-Simulacrum as far as I can see the only part that the language has to support is preventing optimizations - stabilizing black_box
. Everything else can be built on top using external crates. Am I missing something?
Like #[test]
, #[bench]
has special powers not available to #[proc_macro_attribute]
: all functions in a crate with these attributes are "collected" and then some code is generated at the crate top-level to call them all.
black_box is separately tracked by https://github.com/rust-lang/rust/issues/64102 these days and is not necessary for a baseline stabilization (the closure return value is already passed through a black box, IIRC).
FWIW, I don't personally see bench as distinct from test there -- they both share nearly all of the same infrastructure in the compiler and libtest. If we wanted to make test the only special attribute and require #[test] #[bench]
or #[test(bench)]
or something on all benchmarks in the stable variant, that would probably be fine (just seems like needless churn to me).
Given that the custom test frameworks experiment has been closed, how does @rust-lang/lang and @rust-lang/libs-api feel about either:
#[bench]
attribute and a minimal Bencher
API as-is (perhaps in a different module than test::bench
?)cargo bench
would continue to run benches/*.rs
to support external tools like criterion.Cargo.toml
changes for a future edition: make harness = false
the default in [[benches]]
sections, make bench = false
the default in other "target" sections ([lib]
, [[bin]]
, [[example]]
, and [[test]]
).
This is a tracking issue for the
#[bench]
attribute and its stability in the compiler. Currently it is not possible to use this from stable Rust as it requiresextern crate test
which is itself not stable.Core APIs for benchmarking:
#[bench]
, which means the function should take a&mut Bencher
argument and will be run as part of regular tests (and specifically benchmarked with --bench passed to the test binary).