search

lambdaclass / cairo-vm

cairo-vm is a Rust implementation of the Cairo VM. Cairo (CPU Algebraic Intermediate Representation) is a programming language for writing provable programs, where one party can prove to another that a certain computation was executed correctly without the need for this party to re-execute the same program.
https://lambdaclass.github.io/cairo-vm
Apache License 2.0
488 stars 133 forks source link

Add Hyper Threading benchmarks workflow #1682

Closed pefontana closed 3 months ago

pefontana commented 4 months ago

Add Hyper Threading benchmarks workflow

Description

Description of the pull request changes and motivation.

Checklist

codecov[bot] commented 4 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 94.98%. Comparing base (635fef9) to head (4e6d75b).

:exclamation: Current head 4e6d75b differs from pull request most recent head 502017e. Consider uploading reports for the commit 502017e to get more accurate results

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1682 +/- ## ========================================== - Coverage 94.98% 94.98% -0.01% ========================================== Files 99 99 Lines 38580 38579 -1 ========================================== - Hits 36647 36646 -1 Misses 1933 1933 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

github-actions[bot] commented 4 months ago

Benchmark Results for unmodified programs :rocket:

Command Mean [s] Min [s] Max [s] Relative
base big_factorial 2.284 ± 0.014 2.254 2.297 1.00 ± 0.01
head big_factorial 2.279 ± 0.012 2.264 2.295 1.00
Command Mean [s] Min [s] Max [s] Relative
base big_fibonacci 2.275 ± 0.017 2.241 2.289 1.00
head big_fibonacci 2.297 ± 0.027 2.239 2.320 1.01 ± 0.01
Command Mean [s] Min [s] Max [s] Relative
base blake2s_integration_benchmark 8.557 ± 0.186 8.315 8.820 1.00 ± 0.03
head blake2s_integration_benchmark 8.518 ± 0.208 8.325 8.893 1.00
Command Mean [s] Min [s] Max [s] Relative
base compare_arrays_200000 2.340 ± 0.013 2.330 2.364 1.01 ± 0.01
head compare_arrays_200000 2.322 ± 0.013 2.308 2.343 1.00
Command Mean [s] Min [s] Max [s] Relative
base dict_integration_benchmark 1.447 ± 0.018 1.425 1.487 1.00
head dict_integration_benchmark 1.457 ± 0.008 1.449 1.474 1.01 ± 0.01
Command Mean [s] Min [s] Max [s] Relative
base field_arithmetic_get_square_benchmark 1.320 ± 0.040 1.289 1.411 1.01 ± 0.03
head field_arithmetic_get_square_benchmark 1.311 ± 0.014 1.288 1.327 1.00
Command Mean [s] Min [s] Max [s] Relative
base integration_builtins 8.489 ± 0.167 8.323 8.770 1.00
head integration_builtins 8.542 ± 0.184 8.317 8.730 1.01 ± 0.03
Command Mean [s] Min [s] Max [s] Relative
base keccak_integration_benchmark 9.013 ± 0.818 8.567 11.288 1.02 ± 0.10
head keccak_integration_benchmark 8.827 ± 0.217 8.559 9.010 1.00
Command Mean [s] Min [s] Max [s] Relative
base linear_search 2.372 ± 0.036 2.341 2.456 1.01 ± 0.02
head linear_search 2.349 ± 0.015 2.326 2.377 1.00
Command Mean [s] Min [s] Max [s] Relative
base math_cmp_and_pow_integration_benchmark 1.590 ± 0.016 1.567 1.611 1.01 ± 0.01
head math_cmp_and_pow_integration_benchmark 1.576 ± 0.015 1.555 1.611 1.00
Command Mean [s] Min [s] Max [s] Relative
base math_integration_benchmark 1.436 ± 0.028 1.410 1.507 1.00 ± 0.02
head math_integration_benchmark 1.432 ± 0.018 1.413 1.467 1.00
Command Mean [s] Min [s] Max [s] Relative
base memory_integration_benchmark 1.277 ± 0.020 1.257 1.319 1.00 ± 0.02
head memory_integration_benchmark 1.276 ± 0.017 1.257 1.313 1.00
Command Mean [s] Min [s] Max [s] Relative
base operations_with_data_structures_benchmarks 1.620 ± 0.021 1.604 1.674 1.00
head operations_with_data_structures_benchmarks 1.625 ± 0.032 1.603 1.697 1.00 ± 0.02
Command Mean [ms] Min [ms] Max [ms] Relative
base pedersen 593.7 ± 2.8 590.5 600.0 1.00
head pedersen 595.7 ± 4.9 589.8 603.6 1.00 ± 0.01
Command Mean [ms] Min [ms] Max [ms] Relative
base poseidon_integration_benchmark 978.4 ± 7.0 967.3 985.8 1.01 ± 0.01
head poseidon_integration_benchmark 972.5 ± 6.4 960.8 982.0 1.00
Command Mean [s] Min [s] Max [s] Relative
base secp_integration_benchmark 1.878 ± 0.023 1.860 1.935 1.00 ± 0.02
head secp_integration_benchmark 1.877 ± 0.025 1.857 1.941 1.00
Command Mean [ms] Min [ms] Max [ms] Relative
base set_integration_benchmark 724.1 ± 7.8 718.3 744.6 1.00 ± 0.01
head set_integration_benchmark 720.5 ± 2.9 715.7 725.5 1.00
Command Mean [s] Min [s] Max [s] Relative
base uint256_integration_benchmark 4.648 ± 0.061 4.584 4.745 1.00
head uint256_integration_benchmark 4.752 ± 0.136 4.590 5.100 1.02 ± 0.03
github-actions[bot] commented 3 months ago

Hyper Thereading Benchmark results hyperfine -r 2 -n "hyper_threading_main threads: 1" 'RAYON_NUM_THREADS=1 ./hyper_threading_main' -n "hyper_threading_pr threads: 1" 'RAYON_NUM_THREADS=1 ./hyper_threading_pr' n Benchmark 1: hyper_threading_main threads: 1 Time (mean ± σ): 29.359 s ± 0.046 s [User: 28.693 s, System: 0.663 s] Range (min … max): 29.327 s … 29.392 s 2 runs

Benchmark 2: hyper_threading_pr threads: 1 Time (mean ± σ): 29.415 s ± 0.097 s [User: 28.712 s, System: 0.701 s] Range (min … max): 29.347 s … 29.484 s 2 runs

Summary 'hyper_threading_main threads: 1' ran 1.00 ± 0.00 times faster than 'hyper_threading_pr threads: 1' Hyper Thereading Benchmark results hyperfine -r 2 -n "hyper_threading_main threads: 2" 'RAYON_NUM_THREADS=2 ./hyper_threading_main' -n "hyper_threading_pr threads: 2" 'RAYON_NUM_THREADS=2 ./hyper_threading_pr' n Benchmark 1: hyper_threading_main threads: 2 Time (mean ± σ): 15.662 s ± 0.025 s [User: 28.711 s, System: 0.708 s] Range (min … max): 15.644 s … 15.680 s 2 runs

Benchmark 2: hyper_threading_pr threads: 2 Time (mean ± σ): 15.765 s ± 0.072 s [User: 28.870 s, System: 0.764 s] Range (min … max): 15.714 s … 15.816 s 2 runs

Summary 'hyper_threading_main threads: 2' ran 1.01 ± 0.00 times faster than 'hyper_threading_pr threads: 2' Hyper Thereading Benchmark results hyperfine -r 2 -n "hyper_threading_main threads: 4" 'RAYON_NUM_THREADS=4 ./hyper_threading_main' -n "hyper_threading_pr threads: 4" 'RAYON_NUM_THREADS=4 ./hyper_threading_pr' n Benchmark 1: hyper_threading_main threads: 4 Time (mean ± σ): 11.452 s ± 0.327 s [User: 41.068 s, System: 0.863 s] Range (min … max): 11.221 s … 11.683 s 2 runs

Benchmark 2: hyper_threading_pr threads: 4 Time (mean ± σ): 11.444 s ± 0.245 s [User: 40.916 s, System: 0.930 s] Range (min … max): 11.271 s … 11.617 s 2 runs

Summary 'hyper_threading_pr threads: 4' ran 1.00 ± 0.04 times faster than 'hyper_threading_main threads: 4' Hyper Thereading Benchmark results hyperfine -r 2 -n "hyper_threading_main threads: 6" 'RAYON_NUM_THREADS=6 ./hyper_threading_main' -n "hyper_threading_pr threads: 6" 'RAYON_NUM_THREADS=6 ./hyper_threading_pr' n Benchmark 1: hyper_threading_main threads: 6 Time (mean ± σ): 11.306 s ± 0.045 s [User: 41.214 s, System: 0.978 s] Range (min … max): 11.275 s … 11.338 s 2 runs

Benchmark 2: hyper_threading_pr threads: 6 Time (mean ± σ): 11.771 s ± 0.044 s [User: 41.164 s, System: 0.897 s] Range (min … max): 11.740 s … 11.802 s 2 runs

Summary 'hyper_threading_main threads: 6' ran 1.04 ± 0.01 times faster than 'hyper_threading_pr threads: 6' Hyper Thereading Benchmark results hyperfine -r 2 -n "hyper_threading_main threads: 8" 'RAYON_NUM_THREADS=8 ./hyper_threading_main' -n "hyper_threading_pr threads: 8" 'RAYON_NUM_THREADS=8 ./hyper_threading_pr' n Benchmark 1: hyper_threading_main threads: 8 Time (mean ± σ): 11.038 s ± 0.049 s [User: 41.519 s, System: 1.021 s] Range (min … max): 11.003 s … 11.073 s 2 runs

Benchmark 2: hyper_threading_pr threads: 8 Time (mean ± σ): 11.064 s ± 0.011 s [User: 41.569 s, System: 0.931 s] Range (min … max): 11.057 s … 11.072 s 2 runs

Summary 'hyper_threading_main threads: 8' ran 1.00 ± 0.00 times faster than 'hyper_threading_pr threads: 8' Hyper Thereading Benchmark results hyperfine -r 2 -n "hyper_threading_main threads: 16" 'RAYON_NUM_THREADS=16 ./hyper_threading_main' -n "hyper_threading_pr threads: 16" 'RAYON_NUM_THREADS=16 ./hyper_threading_pr' n Benchmark 1: hyper_threading_main threads: 16 Time (mean ± σ): 11.145 s ± 0.229 s [User: 41.702 s, System: 1.069 s] Range (min … max): 10.983 s … 11.307 s 2 runs

Benchmark 2: hyper_threading_pr threads: 16 Time (mean ± σ): 11.169 s ± 0.032 s [User: 41.595 s, System: 1.007 s] Range (min … max): 11.147 s … 11.192 s 2 runs

Summary 'hyper_threading_main threads: 16' ran 1.00 ± 0.02 times faster than 'hyper_threading_pr threads: 16'

github-actions[bot] commented 3 months ago 
**Hyper Thereading Benchmark results**
\n \n \n 
\n \n \n 
hyperfine -r 2 -n "hyper_threading_main threads: 2" 'RAYON_NUM_THREADS=2 ./hyper_threading_main' -n "hyper_threading_pr threads: 2" 'RAYON_NUM_THREADS=2 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 2
  Time (mean ± σ):     15.742 s ±  0.072 s    [User: 28.926 s, System: 0.744 s]
  Range (min … max):   15.691 s … 15.793 s    2 runs

Benchmark 2: hyper_threading_pr threads: 2
  Time (mean ± σ):     15.830 s ±  0.009 s    [User: 28.989 s, System: 0.704 s]
  Range (min … max):   15.824 s … 15.836 s    2 runs

Summary
  'hyper_threading_main threads: 2' ran
    1.01 ± 0.00 times faster than 'hyper_threading_pr threads: 2'
\n \n \n 
\n \n \n 
hyperfine -r 2 -n "hyper_threading_main threads: 4" 'RAYON_NUM_THREADS=4 ./hyper_threading_main' -n "hyper_threading_pr threads: 4" 'RAYON_NUM_THREADS=4 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 4
  Time (mean ± σ):     11.311 s ±  0.350 s    [User: 41.077 s, System: 0.817 s]
  Range (min … max):   11.064 s … 11.559 s    2 runs

Benchmark 2: hyper_threading_pr threads: 4
  Time (mean ± σ):     11.182 s ±  0.003 s    [User: 41.165 s, System: 0.877 s]
  Range (min … max):   11.180 s … 11.184 s    2 runs

Summary
  'hyper_threading_pr threads: 4' ran
    1.01 ± 0.03 times faster than 'hyper_threading_main threads: 4'
\n \n \n