Benchmark within crate source code isn't running

[vlovich]

I'm trying out Divan on Linux. My project is a workspace layout. Within that workspace I have a crate foo. Within foo/benches I create a foo.rs file that has:

fn main() { eprintln!("Benchmark main"); divan::main() }

and register it within foo/Cargo.toml:

[[benches]]
name = "foo"
harness = false

Then within foo/src/lib.rs I put:

#[divan::bench]
pub fn bench() {
  eprintln!("Running registered benchmark");
}

I see "Benchmark main" when I run cargo bench -p foo but not "Running registered benchmark". If I move the bench function to foo/benches/foo.rs then everything works. Not sure what I need to do to make this work.

Semi-related, does putting benchmark code within the crate mean that downstream dependencies compile the benchmarks? I'm hoping it at least is dead-code stripped but would be good to call this out in the docs.

[nvzqz]

Might be that you're not actually using anything from the lib and thus it gets completely removed from your dependency tree. Can you please check if either of the following fixes this?

• Use an item from the library within your benchmarks
• Emit #[used(linker)] in the proc macro

Semi-related, does putting benchmark code within the crate mean that downstream dependencies compile the benchmarks?

Yes. That is why Divan has an internal_benches feature flag. See https://github.com/nvzqz/divan/issues/2#issuecomment-1785991600 for details.

I'm hoping it at least is dead-code stripped but would be good to call this out in the docs.

Dead code elimination doesn't apply here because the approaches used (pre-main and linkme) each have the symbols be required to exist in the final binary regardless of whether they're used.

[vlovich]

Ah, yes. I'm probably not using anything from the lib. Will try again.

FWIW the workaround I'm using re feature flag is to have the library have a dev-dependency on itself with the benchmark feature enabled. It does mean that benchmarks get compiled into tests but that's generally a small price to pay considering that it does eliminate the benchmark support code for external users. Probably the best that can be done at this time until Cargo/Rust improve the benchmarking ecosystem.

[vlovich]

Here's all I needed to change to make it work:

Cargo.toml:

[dependencies]
divan = { version = "0.1", optional = true }

[dev-dependencies]
my-crate = { path = ".", features = ["internal-benchmarks"] }

[features]
internal-benchmarks = ["dep:divan"]

lib.rs:

#[cfg(feature = "internal-benchmarks")]
pub const CRATE_USED: bool = true;

#[cfg(feature = "internal-benchmarks")]
mod benchmarks {
  #[divan::bench]
  fn my_benchmark() {}
}

in benchmark main:

fn main() {
    let _ = memcache::CRATE_USED;
    divan::main();
}

Not the best setup because it forces dependencies needed for running the benchmark itself to be in the [dependencies] section (I mitigate that impact by making them optional & only imported by the internal-benchmarks feature.

[Easyoakland]

@vlovich In my tests it seems that you don't need to create the CRATE_USED const. It's enough to use my_crate; or extern crate my-crate; at the top of the file with the fn main() {divan::main()}