search

tokio-rs / tokio

A runtime for writing reliable asynchronous applications with Rust. Provides I/O, networking, scheduling, timers, ...
https://tokio.rs
MIT License
26.98k stars 2.48k forks source link

Run benchmarks in CI #6539

Open Darksonn opened 6 months ago

Darksonn commented 6 months ago

We don't currently run benchmarks in CI, because doing that without it being really flaky is hard. However, if we used a tool such as iai-callgrind to get consistent measurements using something other than time, then this would become feasible.

Doing this would let us more easily catch performance regressions. Inspired by Towards Impeccable Rust by Jon Gjengset.

wathenjiang commented 6 months ago

Unfortunately, Valgrind's Callgrind tool does not support parallel applications like in production environments. Please refer to: https://valgrind.org/docs/manual/manual-core.html#manual-core.pthreads

This means that once the benchmark of the application is related to multithreading, using the Callgrind tool may not provide valuable benchmark results. Unfortunately, currently most benchmark tests in the benches/ are under multi threads.

I think before considering whether iai-callgrind should be introduced into CI, we should first identify which code blocks are suitable for Callgrind based benchmark.