search

lambdaclass / cairo-vm

cairo-vm is a Rust implementation of the Cairo VM. Cairo (CPU Algebraic Intermediate Representation) is a programming language for writing provable programs, where one party can prove to another that a certain computation was executed correctly without the need for this party to re-execute the same program.
https://lambdaclass.github.io/cairo-vm
Apache License 2.0
485 stars 133 forks source link

Add Hyper Threading benchmarks workflow #1711

Closed pefontana closed 2 months ago

pefontana commented 2 months ago

Add Hyper Threading benchmarks workflow

Description

Add a workflow to benchmark the Cairo-vm in a Hyper-Threading context

Checklist

github-actions[bot] commented 2 months ago 
**Hyper Thereading Benchmark results**

hyperfine -r 2 -n "hyper_threading_main threads: 1" 'RAYON_NUM_THREADS=1 ./hyper_threading_main' -n "hyper_threading_pr threads: 1" 'RAYON_NUM_THREADS=1 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 1
  Time (mean ± σ):     29.270 s ±  0.046 s    [User: 28.600 s, System: 0.666 s]
  Range (min … max):   29.237 s … 29.302 s    2 runs

Benchmark 2: hyper_threading_pr threads: 1
  Time (mean ± σ):     29.345 s ±  0.039 s    [User: 28.648 s, System: 0.694 s]
  Range (min … max):   29.318 s … 29.373 s    2 runs

Summary
  'hyper_threading_main threads: 1' ran
    1.00 ± 0.00 times faster than 'hyper_threading_pr threads: 1'

hyperfine -r 2 -n "hyper_threading_main threads: 2" 'RAYON_NUM_THREADS=2 ./hyper_threading_main' -n "hyper_threading_pr threads: 2" 'RAYON_NUM_THREADS=2 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 2
  Time (mean ± σ):     15.714 s ±  0.021 s    [User: 28.777 s, System: 0.744 s]
  Range (min … max):   15.699 s … 15.729 s    2 runs

Benchmark 2: hyper_threading_pr threads: 2
  Time (mean ± σ):     15.741 s ±  0.031 s    [User: 28.914 s, System: 0.674 s]
  Range (min … max):   15.719 s … 15.763 s    2 runs

Summary
  'hyper_threading_main threads: 2' ran
    1.00 ± 0.00 times faster than 'hyper_threading_pr threads: 2'

hyperfine -r 2 -n "hyper_threading_main threads: 4" 'RAYON_NUM_THREADS=4 ./hyper_threading_main' -n "hyper_threading_pr threads: 4" 'RAYON_NUM_THREADS=4 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 4
  Time (mean ± σ):     11.217 s ±  0.203 s    [User: 40.129 s, System: 0.907 s]
  Range (min … max):   11.073 s … 11.360 s    2 runs

Benchmark 2: hyper_threading_pr threads: 4
  Time (mean ± σ):     11.407 s ±  0.376 s    [User: 40.526 s, System: 0.934 s]
  Range (min … max):   11.141 s … 11.673 s    2 runs

Summary
  'hyper_threading_main threads: 4' ran
    1.02 ± 0.04 times faster than 'hyper_threading_pr threads: 4'

hyperfine -r 2 -n "hyper_threading_main threads: 6" 'RAYON_NUM_THREADS=6 ./hyper_threading_main' -n "hyper_threading_pr threads: 6" 'RAYON_NUM_THREADS=6 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 6
  Time (mean ± σ):     11.056 s ±  0.040 s    [User: 40.792 s, System: 0.923 s]
  Range (min … max):   11.027 s … 11.085 s    2 runs

Benchmark 2: hyper_threading_pr threads: 6
  Time (mean ± σ):     11.341 s ±  0.083 s    [User: 40.571 s, System: 0.945 s]
  Range (min … max):   11.282 s … 11.400 s    2 runs

Summary
  'hyper_threading_main threads: 6' ran
    1.03 ± 0.01 times faster than 'hyper_threading_pr threads: 6'

hyperfine -r 2 -n "hyper_threading_main threads: 8" 'RAYON_NUM_THREADS=8 ./hyper_threading_main' -n "hyper_threading_pr threads: 8" 'RAYON_NUM_THREADS=8 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 8
  Time (mean ± σ):     10.978 s ±  0.346 s    [User: 40.818 s, System: 0.986 s]
  Range (min … max):   10.733 s … 11.222 s    2 runs

Benchmark 2: hyper_threading_pr threads: 8
  Time (mean ± σ):     11.066 s ±  0.027 s    [User: 41.438 s, System: 0.901 s]
  Range (min … max):   11.047 s … 11.085 s    2 runs

Summary
  'hyper_threading_main threads: 8' ran
    1.01 ± 0.03 times faster than 'hyper_threading_pr threads: 8'

hyperfine -r 2 -n "hyper_threading_main threads: 16" 'RAYON_NUM_THREADS=16 ./hyper_threading_main' -n "hyper_threading_pr threads: 16" 'RAYON_NUM_THREADS=16 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 16
  Time (mean ± σ):     11.301 s ±  0.225 s    [User: 41.146 s, System: 0.998 s]
  Range (min … max):   11.142 s … 11.460 s    2 runs

Benchmark 2: hyper_threading_pr threads: 16
  Time (mean ± σ):     11.163 s ±  0.054 s    [User: 42.116 s, System: 0.985 s]
  Range (min … max):   11.125 s … 11.201 s    2 runs

Summary
  'hyper_threading_pr threads: 16' ran
    1.01 ± 0.02 times faster than 'hyper_threading_main threads: 16'
codecov[bot] commented 2 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 94.90%. Comparing base (e629428) to head (a005262).

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1711 +/- ## ======================================= Coverage 94.90% 94.90% ======================================= Files 98 98 Lines 38092 38092 ======================================= Hits 36151 36151 Misses 1941 1941 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

github-actions[bot] commented 2 months ago

Benchmark Results for unmodified programs :rocket:

Command Mean [s] Min [s] Max [s] Relative
base big_factorial 2.299 ± 0.022 2.282 2.357 1.00 ± 0.01
head big_factorial 2.297 ± 0.020 2.277 2.345 1.00
Command Mean [s] Min [s] Max [s] Relative
base big_fibonacci 2.287 ± 0.019 2.264 2.326 1.00
head big_fibonacci 2.314 ± 0.055 2.276 2.454 1.01 ± 0.03
Command Mean [s] Min [s] Max [s] Relative
base blake2s_integration_benchmark 8.584 ± 0.171 8.364 8.732 1.00
head blake2s_integration_benchmark 8.695 ± 0.104 8.406 8.775 1.01 ± 0.02
Command Mean [s] Min [s] Max [s] Relative
base compare_arrays_200000 2.364 ± 0.069 2.327 2.557 1.00 ± 0.03
head compare_arrays_200000 2.362 ± 0.024 2.333 2.413 1.00
Command Mean [s] Min [s] Max [s] Relative
base dict_integration_benchmark 1.472 ± 0.011 1.456 1.488 1.00 ± 0.01
head dict_integration_benchmark 1.468 ± 0.012 1.453 1.487 1.00
Command Mean [s] Min [s] Max [s] Relative
base field_arithmetic_get_square_benchmark 1.397 ± 0.012 1.380 1.418 1.00
head field_arithmetic_get_square_benchmark 1.413 ± 0.019 1.391 1.441 1.01 ± 0.02
Command Mean [s] Min [s] Max [s] Relative
base integration_builtins 8.636 ± 0.273 8.347 9.285 1.02 ± 0.04
head integration_builtins 8.459 ± 0.158 8.340 8.692 1.00
Command Mean [s] Min [s] Max [s] Relative
base keccak_integration_benchmark 8.785 ± 0.169 8.635 9.001 1.00 ± 0.03
head keccak_integration_benchmark 8.761 ± 0.187 8.602 8.999 1.00
Command Mean [s] Min [s] Max [s] Relative
base linear_search 2.383 ± 0.021 2.356 2.421 1.00 ± 0.02
head linear_search 2.383 ± 0.032 2.353 2.457 1.00
Command Mean [s] Min [s] Max [s] Relative
base math_cmp_and_pow_integration_benchmark 1.834 ± 0.021 1.812 1.871 1.00 ± 0.01
head math_cmp_and_pow_integration_benchmark 1.826 ± 0.015 1.802 1.847 1.00
Command Mean [s] Min [s] Max [s] Relative
base math_integration_benchmark 1.617 ± 0.015 1.599 1.643 1.00
head math_integration_benchmark 1.636 ± 0.018 1.611 1.670 1.01 ± 0.01
Command Mean [s] Min [s] Max [s] Relative
base memory_integration_benchmark 1.283 ± 0.015 1.269 1.317 1.00
head memory_integration_benchmark 1.287 ± 0.009 1.273 1.299 1.00 ± 0.01
Command Mean [s] Min [s] Max [s] Relative
base operations_with_data_structures_benchmarks 1.922 ± 0.006 1.913 1.929 1.00
head operations_with_data_structures_benchmarks 1.933 ± 0.024 1.920 2.000 1.01 ± 0.01
Command Mean [ms] Min [ms] Max [ms] Relative
base pedersen 547.1 ± 1.7 545.0 550.3 1.00
head pedersen 548.7 ± 4.2 544.0 559.1 1.00 ± 0.01
Command Mean [ms] Min [ms] Max [ms] Relative
base poseidon_integration_benchmark 977.6 ± 8.0 969.2 996.9 1.00
head poseidon_integration_benchmark 982.8 ± 8.5 968.9 996.3 1.01 ± 0.01
Command Mean [s] Min [s] Max [s] Relative
base secp_integration_benchmark 1.928 ± 0.021 1.914 1.978 1.00 ± 0.01
head secp_integration_benchmark 1.926 ± 0.015 1.908 1.962 1.00
Command Mean [ms] Min [ms] Max [ms] Relative
base set_integration_benchmark 746.0 ± 2.4 743.3 751.4 1.00
head set_integration_benchmark 748.1 ± 5.0 742.4 758.1 1.00 ± 0.01
Command Mean [s] Min [s] Max [s] Relative
base uint256_integration_benchmark 4.685 ± 0.051 4.643 4.763 1.00
head uint256_integration_benchmark 4.711 ± 0.056 4.639 4.763 1.01 ± 0.02