signalapp / libsignal

Home to the Signal Protocol as well as other cryptographic primitives which make Signal possible.
GNU Affero General Public License v3.0
3.08k stars 362 forks source link

Update Benchmarks to include warmup and higher sample size #457

Closed infosechoudini closed 2 years ago

infosechoudini commented 2 years ago

In the Rust portion of the repo under benches, we should implement criterion in a way that allows a higher warmup of the CPU (3s to 5s) before sampling. In addition, we should increase the sample size from 100 -> 150 and bootstrap samples to 100,000 to have a better idea of deviations.

rust/protocol/benches/session.rs

// Needed for Criterion Warmup
use std::time::Duration;

 // Old Benchmark Code Here //

// remove the following
//criterion_group!(benches, session_encrypt, session_encrypt_decrypt);

criterion_group!(
    name = benches;
    config = Criterion::default()
        .warm_up_time(Duration::from_secs(5))
        .measurement_time(Duration::from_secs(10))
        // degree of noise to ignore in measurements, here 1%
        .noise_threshold(0.01)
        // likelihood of noise registering as difference, here 5%
        .significance_level(0.05)
        // likelihood of capturing the true runtime, here 95%
        .confidence_level(0.95)
        // total number of bootstrap resamples, higher is less noisy but slower
        .nresamples(100_000)
        // total samples to collect within the set measurement time
        .sample_size(150);
    targets = session_encrypt, session_encrypt_decrypt
);

criterion_main!(benches);