Error: unable to open connection to bench target

[CAD97]

D:\repos\cad97\playground> cargo criterion      
    Finished bench [optimized] target(s) in 0.19s
Error: Unable to open connection to bench target main

Caused by:
Gnuplot not found, using plotters backend
    A non-blocking socket operation could not be completed immediately. (os error 10035)thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: IoError(Os { code: 10054, kind: ConnectionReset, message: "An existing connection was forcibly closed by the remote host." })

', Stack backtrace:
D:\.rust\cargo\registry\src\github.com-1ecc6299db9ec823\criterion-0.3.4\src\lib.rs   0: std::backtrace_rs::backtrace::dbghelp::trace
             at /rustc/e37a13cc3594004663738bd18d8100e6db9666cf\/library\std\src\..\..\backtrace\src\backtrace\dbghelp.rs:98
   1: std::backtrace_rs::backtrace::trace_unsynchronized
             at /rustc/e37a13cc3594004663738bd18d8100e6db9666cf\/library\std\src\..\..\backtrace\src\backtrace\mod.rs:66
   2: std::backtrace::Backtrace::create
             at /rustc/e37a13cc3594004663738bd18d8100e6db9666cf\/library\std\src\backtrace.rs:327
   3: std::backtrace::Backtrace::capture
             at /rustc/e37a13cc3594004663738bd18d8100e6db9666cf\/library\std\src\backtrace.rs:295
   4: cargo_criterion::connection::Connection::new
   5: cargo_criterion::bench_target::BenchTarget::execute
   6: cargo_criterion::stats::bivariate::Data<X,Y>::bootstrap
   7: std::sys_common::backtrace::__rust_begin_short_backtrace
   8: std::rt::lang_start::{{closure}}
   9: core::ops::function::impls::{{impl}}::call_once
             at /rustc/e37a13cc3594004663738bd18d8100e6db9666cf\library\core\src\ops\function.rs:280
  10: std::panicking::try::do_call
             at /rustc/e37a13cc3594004663738bd18d8100e6db9666cf\/library\std\src\panicking.rs:379
  11: std::panicking::try
             at /rustc/e37a13cc3594004663738bd18d8100e6db9666cf\/library\std\src\panicking.rs:343
  12: std::panic::catch_unwind
             at /rustc/e37a13cc3594004663738bd18d8100e6db9666cf\/library\std\src\panic.rs:431
  13: std::rt::lang_start_internal
             at /rustc/e37a13cc3594004663738bd18d8100e6db9666cf\/library\std\src\rt.rs:51
  14: main
  15: invoke_main
             at d:\agent\_work\63\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:78
  16: __scrt_common_main_seh
             at d:\agent\_work\63\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
  17: BaseThreadInitThunk
  18: RtlUserThreadStart:
1105:18
stack backtrace:
   0: std::panicking::begin_panic_handler
             at /rustc/e37a13cc3594004663738bd18d8100e6db9666cf\/library\std\src\panicking.rs:493
   1: core::panicking::panic_fmt
             at /rustc/e37a13cc3594004663738bd18d8100e6db9666cf\/library\core\src\panicking.rs:92
   2: core::option::expect_none_failed
             at /rustc/e37a13cc3594004663738bd18d8100e6db9666cf\/library\core\src\option.rs:1300
   3: criterion::Criterion<M>::benchmark_group
   4: criterion::report::_::<impl serde::ser::Serialize for criterion::report::BenchmarkId>::serialize
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

I'm not sure what happened here, I just installed cargo-criterion and did cargo criterion.

OS: Windows
Toolchain: nightly-x86_64-pc-windows-msvc (directory override for '\?\D:\repos\cad97\playground')
rustc 1.52.0-nightly (e37a13cc3 2021-02-28)
Version: cargo-criterion 1.0.1
Criterion: 0.3.4

[richlowe]

This is true on other systems too. the cargo-criterion runner uses a non-blocking socket in BenchTarget::execute, and in a lot of other places just uses ? on socket errors, which means that EWOULDBLOCK reaches the top level and panics if ever data is not available in time, and makes the whole thing very racey.

A simple workaround is to remove the code in that method that sets the listener non-blocking, though I'm not sure if there are any downsides.

[seigert]

I have the same problem under macOS 11.2.3, rustc/cargo 1.50.0, cargo-criterion 1.0.1, criterion 0.3.4.

It's even reproducible with a simple benchmark as this:

use criterion::{criterion_group, criterion_main, BatchSize, BenchmarkId, Criterion, Throughput};
use rand::RngCore;

fn int_add_benchmark(c: &mut Criterion) {
    let mut rng = rand::thread_rng();
    let mut group = c.benchmark_group("add");

    let x = rng.next_u32();
    let y = rng.next_u32();

    let pair = (&x, &y);
    group.bench_with_input(
        BenchmarkId::new("x & y", format!("({}, {})", &x, &y)),
        &pair,
        |b, &(x, y)| b.iter(|| x + y),
    );

    group.finish();
}

criterion_group!(add_benches, int_add_benchmark,);
criterion_main!(add_benches,);

[domenicquirl]

Bumping for visibility - I just ran into this on Windows installing cargo criterion for the first time after using criterion benchmarks for a while. The error occurs every time, including for any individual bench targeted with --bench.

Toolchain: stable-x86_64-pc-windows-msvc rustc: 1.52.1 (9bc8c42bb 2021-05-09) cargo: 1.52.0 (69767412a 2021-04-21) cargo-criterion: 1.0.1 criterion: 0.3.3

The traces I get seem to be a bit different:

Backtrace

``` 15: 0x7ff66afb3193 - core::option::expect_none_failed at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53\/library\core\src\option.rs:1329 16: 0x7ff66ad469cd - criterion::Criterion::benchmark_group::hd7ba2b0f40564016 17: 0x7ff66ad3ecf1 - criterion::estimate::_::::serialize::haf85b50779ff9c4a 18: 0x7ff66ad598c6 - std::sys_common::backtrace::__rust_begin_short_backtrace::h444449e07998601a 19: 0x7ff66ad405fc - std::rt::lang_start::{{closure}}::h5cee52f6a8bc9e2b 20: 0x7ff66af847f7 - core::ops::function::impls::{{impl}}::call_once at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53\library\core\src\ops\function.rs:280 21: 0x7ff66af847f7 - std::panicking::try::do_call at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53\/library\std\src\panicking.rs:379 22: 0x7ff66af847f7 - std::panicking::try at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53\/library\std\src\panicking.rs:343 23: 0x7ff66af847f7 - std::panic::catch_unwind at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53\/library\std\src\panic.rs:431 ```

Backtrace for individual `--bench`

``` 15: 0x7ff7c98c2ee3 - core::option::expect_none_failed at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53\/library\core\src\option.rs:1329 16: 0x7ff7c963289d - criterion::Criterion::benchmark_group::hc664376cd10beb13 17: 0x7ff7c962ad2e - criterion::stats::univariate::percentiles::Percentiles::quartiles::hfc1d0c94ba4a5c23 18: 0x7ff7c9626496 - std::sys_common::backtrace::__rust_begin_short_backtrace::hc62009e4fd2377e4 19: 0x7ff7c95d67dc - std::rt::lang_start::{{closure}}::h4a383f55e15bb8e8 20: 0x7ff7c988c327 - core::ops::function::impls::{{impl}}::call_once at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53\library\core\src\ops\function.rs:280 21: 0x7ff7c988c327 - std::panicking::try::do_call at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53\/library\std\src\panicking.rs:379 22: 0x7ff7c988c327 - std::panicking::try at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53\/library\std\src\panicking.rs:343 23: 0x7ff7c988c327 - std::panic::catch_unwind at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53\/library\std\src\panic.rs:431 ```

Fixed in cargo-criterion-1.1.0.