search

bheisler / criterion.rs

Statistics-driven benchmarking library for Rust
Apache License 2.0
4.52k stars 301 forks source link

[Question] Any idea to support multi threaded routine call for throughput test? #752

Open ileixe opened 8 months ago

ileixe commented 8 months ago

I'm looking for basic and simple throughput test (stress or loadgen) solution which measure goodput and badput. Currently criterion.rs provides simple throughput metrics which single threaded one but what I want multi-threaded form with more complex workloads user can define.

I searched several rust projects and they have somehow similar approach to measure throughput, to generate "workload" from multi threaded on their own (own main function without harness) or more wider form of test theory like goose. Both approaches looks fine but have some limitations (e.g. Goose looks targeting loadgen for "very specific" workload like HTTP) I think it's very helpful if criterion.rs provides basic API for such scenario.

From implementation perspective, now I can use iter_custom() at least as I can define any function, but want to hear better idea.

Any idea would be appreciated.

Pr0methean commented 8 months ago

A way to accomplish this would be if we could specify how to calculate throughput at the end of a batch of iterations, since then the threads that weren't being directly benchmarked could increment an AtomicU64 each time they finished a work unit. Then we'd call swap(0, SeqCst) on that counter at the end of the batch, and store(0, SeqCst) once we'd joined all those threads.