Open andrewgazelka opened 3 months ago
This is definitely on my plans.
@andrewgazelka can you please share what kind of asynchronous code you are testing? These days I'm spending more time testing low-level code and optimizing hot-loops and not so much with benchmarking async code. Some feedback on real world scenarios will help me to create more convenient and flexible API for async case.
@andrewgazelka can you please share what kind of asynchronous code you are testing? These days I'm spending more time testing low-level code and optimizing hot-loops and not so much with benchmarking async code. Some feedback on real world scenarios will help me to create more convenient and flexible API for async case.
async fn throughput() {
let client = tokio::spawn(async move {
let mut client = hyperion_proxy::client::Client::connect("127.0.0.1:8080")
.await
.unwrap();
let request_player_id = RequestIdResponse { id: 1 };
client.send(request_player_id).await.unwrap();
});
client.await.unwrap();
}
fn benchmarks() -> impl IntoBenchmarks {
let runtime = tokio::runtime::Builder::new_multi_thread()
.enable_all()
.build()
.unwrap();
[benchmark_fn("throughput", move |b| {
let handle_runtime = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.unwrap();
runtime.block_on(async move {
let server = tokio::spawn(async move {
hyperion_proxy::server::init().await.unwrap();
});
// delay 1 second
tokio::time::sleep(std::time::Duration::from_secs(1)).await;
let result = b.iter(move || {
let handle = tokio::spawn(throughput());
handle_runtime.block_on(handle).unwrap();
});
server.abort();
result
})
})]
}
tango_benchmarks!(benchmarks());
tango_main!();
Also oddly when I run this I get:
❯ cargo bench --bench=throughput -- compare
Compiling hyperion-proxy v0.1.0 (/Users/andrewgazelka/Projects/minecraft/hyperion-proxy)
Finished `bench` profile [optimized] target(s) in 0.88s
Running benches/throughput.rs (target/release/deps/throughput-eabfb2c83dfdbfac)
thread 'main' panicked at benches/throughput.rs:38:30:
there is no reactor running, must be called from the context of a Tokio 1.x runtime
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
error: bench failed, to rerun pass `--bench throughput`
but I have changed the code so I can't easily re-test. I'm sure if I use a backtrace it will be obvious.
And I agree with you. I've been using this mostly for testing low-level code. However, it can be annoying to have to code things twice: once with async and once with synchronous. So, it can be nice to test actual async code. I think performance in this case is going to really relate to how I'm allocating a buffer that can be read from if it's the correct size, and a lot of things that just—I wouldn't be getting good data if I tried to not use sockets/etc.
Got some proof-of-concept code. Seems like it's working.
factorial [ 531.5 ns ... 532.1 ns ] +0.11%
factorial_async [ 1.4 us ... 1.4 us ] +0.06%
Need to do some more work to make it usable, though
There is some more or less working code in async-poc
. Async testing can be done like this:
[dev-dependencies]
tango-bench = { git = "https://github.com/bazhenov/tango.git", branch = "async-poc", features = ["async-tokio"] }
use tango_bench::{
async_benchmark_fn, asynchronous::tokio::TokioRuntime, IntoBenchmarks,
};
fn benchmarks() -> impl IntoBenchmarks {
[async_benchmark_fn("async_factorial", TokioRuntime, |b| {
b.iter(|| async { factorial(500).await })
})]
}
There are some more or less working code in
async-poc
. Async testing can be done like this:[dev-dependencies] tango-bench = { git = "https://github.com/bazhenov/tango.git", branch = "async-poc", features = ["async-tokio"] }
use tango_bench::{ async_benchmark_fn, asynchronous::tokio::TokioRuntime, IntoBenchmarks, }; fn benchmarks() -> impl IntoBenchmarks { [async_benchmark_fn("async_factorial", TokioRuntime, |b| { b.iter(|| async { factorial(500).await }) })] }
Awesome, thank you so much for working on this. I will let you know how this goes and I will try to benchmark with this.
Of course we can still bench
async
things right now... it can just require a lot of code.criterion
has built-in support for async functions (if you supply a runtime).https://bheisler.github.io/criterion.rs/book/user_guide/benchmarking_async.html