Support `async`

[andrewgazelka]

Of course we can still bench async things right now... it can just require a lot of code.

criterion has built-in support for async functions (if you supply a runtime).

https://bheisler.github.io/criterion.rs/book/user_guide/benchmarking_async.html

[bazhenov]

This is definitely on my plans.

[bazhenov]

@andrewgazelka can you please share what kind of asynchronous code you are testing? These days I'm spending more time testing low-level code and optimizing hot-loops and not so much with benchmarking async code. Some feedback on real world scenarios will help me to create more convenient and flexible API for async case.

[andrewgazelka]

@andrewgazelka can you please share what kind of asynchronous code you are testing? These days I'm spending more time testing low-level code and optimizing hot-loops and not so much with benchmarking async code. Some feedback on real world scenarios will help me to create more convenient and flexible API for async case.

async fn throughput() {
    let client = tokio::spawn(async move {
        let mut client = hyperion_proxy::client::Client::connect("127.0.0.1:8080")
            .await
            .unwrap();

        let request_player_id = RequestIdResponse { id: 1 };
        client.send(request_player_id).await.unwrap();
    });

    client.await.unwrap();
}

fn benchmarks() -> impl IntoBenchmarks {
    let runtime = tokio::runtime::Builder::new_multi_thread()
        .enable_all()
        .build()
        .unwrap();

    [benchmark_fn("throughput", move |b| {
        let handle_runtime = tokio::runtime::Builder::new_current_thread()
            .enable_all()
            .build()
            .unwrap();

        runtime.block_on(async move {
            let server = tokio::spawn(async move {
                hyperion_proxy::server::init().await.unwrap();
            });

            // delay 1 second
            tokio::time::sleep(std::time::Duration::from_secs(1)).await;

            let result = b.iter(move || {
                let handle = tokio::spawn(throughput());
                handle_runtime.block_on(handle).unwrap();
            });

            server.abort();

            result
        })
    })]
}

tango_benchmarks!(benchmarks());
tango_main!();

Also oddly when I run this I get:

❯ cargo bench --bench=throughput -- compare
   Compiling hyperion-proxy v0.1.0 (/Users/andrewgazelka/Projects/minecraft/hyperion-proxy)
    Finished `bench` profile [optimized] target(s) in 0.88s
     Running benches/throughput.rs (target/release/deps/throughput-eabfb2c83dfdbfac)
thread 'main' panicked at benches/throughput.rs:38:30:
there is no reactor running, must be called from the context of a Tokio 1.x runtime
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
error: bench failed, to rerun pass `--bench throughput`

but I have changed the code so I can't easily re-test. I'm sure if I use a backtrace it will be obvious.

---

And I agree with you. I've been using this mostly for testing low-level code. However, it can be annoying to have to code things twice: once with async and once with synchronous. So, it can be nice to test actual async code. I think performance in this case is going to really relate to how I'm allocating a buffer that can be read from if it's the correct size, and a lot of things that just—I wouldn't be getting good data if I tried to not use sockets/etc.

[bazhenov]

Got some proof-of-concept code. Seems like it's working.

factorial                                          [ 531.5 ns ... 532.1 ns ]      +0.11%
factorial_async                                    [   1.4 us ...   1.4 us ]      +0.06%

Need to do some more work to make it usable, though

[bazhenov]

There is some more or less working code in async-poc. Async testing can be done like this:

[dev-dependencies]
tango-bench = { git = "https://github.com/bazhenov/tango.git", branch = "async-poc", features = ["async-tokio"] }

use tango_bench::{
  async_benchmark_fn, asynchronous::tokio::TokioRuntime, IntoBenchmarks,
};

fn benchmarks() -> impl IntoBenchmarks {
  [async_benchmark_fn("async_factorial", TokioRuntime, |b| {
    b.iter(|| async { factorial(500).await })
  })]
}

[andrewgazelka]

There are some more or less working code in async-poc. Async testing can be done like this:

[dev-dependencies]
tango-bench = { git = "https://github.com/bazhenov/tango.git", branch = "async-poc", features = ["async-tokio"] }

use tango_bench::{
  async_benchmark_fn, asynchronous::tokio::TokioRuntime, IntoBenchmarks,
};

fn benchmarks() -> impl IntoBenchmarks {
  [async_benchmark_fn("async_factorial", TokioRuntime, |b| {
    b.iter(|| async { factorial(500).await })
  })]
}

Awesome, thank you so much for working on this. I will let you know how this goes and I will try to benchmark with this.