search

Chia-Network / chia_rs

Rust crate & wheel with consensus code
Apache License 2.0
20 stars 26 forks source link

Fix benchmark behavior #631

Closed Rigidity closed 2 months ago

Rigidity commented 3 months ago

Results compared to prior to interior mutability:

Gnuplot not found, using plotters backend
Benchmarking bls_cache.aggregate_verify, 0% cache hits: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 51.1s, or reduce sample count to 10.
bls_cache.aggregate_verify, 0% cache hits
                        time:   [512.40 ms 516.72 ms 522.79 ms]
                        change: [-0.1042% +0.7965% +2.0013%] (p = 0.16 > 0.05)
                        No change in performance detected.
Found 9 outliers among 100 measurements (9.00%)
  5 (5.00%) high mild
  4 (4.00%) high severe

Benchmarking bls_cache.aggregate_verify, 10% cache hits: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 47.0s, or reduce sample count to 10.
bls_cache.aggregate_verify, 10% cache hits
                        time:   [464.36 ms 465.94 ms 467.72 ms]
                        change: [-8.6168% -8.2999% -7.9271%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 11 outliers among 100 measurements (11.00%)
  2 (2.00%) high mild
  9 (9.00%) high severe

Benchmarking bls_cache.aggregate_verify, 20% cache hits: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 40.8s, or reduce sample count to 10.
bls_cache.aggregate_verify, 20% cache hits
                        time:   [407.63 ms 408.87 ms 410.40 ms]
                        change: [-19.863% -19.596% -19.288%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  1 (1.00%) high mild
  4 (4.00%) high severe

Benchmarking bls_cache.aggregate_verify, 50% cache hits: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 25.6s, or reduce sample count to 10.
bls_cache.aggregate_verify, 50% cache hits
                        time:   [258.91 ms 259.95 ms 261.08 ms]
                        change: [-49.998% -49.686% -49.347%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  5 (5.00%) high mild

bls_cache.aggregate_verify, 100% cache hits
                        time:   [2.1386 ms 2.1594 ms 2.1925 ms]
                        change: [+3.8251% +5.2293% +7.0844%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high severe

Benchmarking bls_cache.aggregate_verify, no cache: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 26.6s, or reduce sample count to 10.
bls_cache.aggregate_verify, no cache
                        time:   [266.31 ms 267.23 ms 268.25 ms]
                        change: [-1.3208% -0.5508% +0.1486%] (p = 0.14 > 0.05)
                        No change in performance detected.
Found 4 outliers among 100 measurements (4.00%)
  2 (2.00%) high mild
  2 (2.00%) high severe
arvidn commented 2 months ago

I think we're abandoning the target branch and picking the parts we want to keep into separate PRs. I don't believe this issue of the benchmark exists in main