search

lambdaclass / cairo-vm

cairo-vm is a Rust implementation of the Cairo VM. Cairo (CPU Algebraic Intermediate Representation) is a programming language for writing provable programs, where one party can prove to another that a certain computation was executed correctly without the need for this party to re-execute the same program.
https://lambdaclass.github.io/cairo-vm
Apache License 2.0
485 stars 132 forks source link

feat: unify test utils to a single feature #1755

Closed Oppen closed 1 month ago

Oppen commented 1 month ago

These are less used features that should only be deployed to developers' machines and checking each of them on their own takes a lot of CI time. Unifying them makes dev cycles shorter.

Checklist

github-actions[bot] commented 1 month ago 
**Hyper Thereading Benchmark results**

hyperfine -r 2 -n "hyper_threading_main threads: 1" 'RAYON_NUM_THREADS=1 ./hyper_threading_main' -n "hyper_threading_pr threads: 1" 'RAYON_NUM_THREADS=1 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 1
  Time (mean ± σ):     26.633 s ±  0.083 s    [User: 25.883 s, System: 0.749 s]
  Range (min … max):   26.575 s … 26.692 s    2 runs

Benchmark 2: hyper_threading_pr threads: 1
  Time (mean ± σ):     26.313 s ±  0.235 s    [User: 25.521 s, System: 0.790 s]
  Range (min … max):   26.146 s … 26.479 s    2 runs

Summary
  'hyper_threading_pr threads: 1' ran
    1.01 ± 0.01 times faster than 'hyper_threading_main threads: 1'

hyperfine -r 2 -n "hyper_threading_main threads: 2" 'RAYON_NUM_THREADS=2 ./hyper_threading_main' -n "hyper_threading_pr threads: 2" 'RAYON_NUM_THREADS=2 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 2
  Time (mean ± σ):     14.080 s ±  0.030 s    [User: 26.049 s, System: 0.742 s]
  Range (min … max):   14.058 s … 14.101 s    2 runs

Benchmark 2: hyper_threading_pr threads: 2
  Time (mean ± σ):     14.170 s ±  0.040 s    [User: 26.173 s, System: 0.732 s]
  Range (min … max):   14.142 s … 14.199 s    2 runs

Summary
  'hyper_threading_main threads: 2' ran
    1.01 ± 0.00 times faster than 'hyper_threading_pr threads: 2'

hyperfine -r 2 -n "hyper_threading_main threads: 4" 'RAYON_NUM_THREADS=4 ./hyper_threading_main' -n "hyper_threading_pr threads: 4" 'RAYON_NUM_THREADS=4 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 4
  Time (mean ± σ):     10.233 s ±  0.153 s    [User: 37.723 s, System: 0.961 s]
  Range (min … max):   10.125 s … 10.340 s    2 runs

Benchmark 2: hyper_threading_pr threads: 4
  Time (mean ± σ):     10.668 s ±  0.256 s    [User: 38.190 s, System: 0.964 s]
  Range (min … max):   10.487 s … 10.849 s    2 runs

Summary
  'hyper_threading_main threads: 4' ran
    1.04 ± 0.03 times faster than 'hyper_threading_pr threads: 4'

hyperfine -r 2 -n "hyper_threading_main threads: 6" 'RAYON_NUM_THREADS=6 ./hyper_threading_main' -n "hyper_threading_pr threads: 6" 'RAYON_NUM_THREADS=6 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 6
  Time (mean ± σ):     10.486 s ±  0.432 s    [User: 38.330 s, System: 0.945 s]
  Range (min … max):   10.181 s … 10.791 s    2 runs

Benchmark 2: hyper_threading_pr threads: 6
  Time (mean ± σ):     10.199 s ±  0.191 s    [User: 38.154 s, System: 0.877 s]
  Range (min … max):   10.065 s … 10.334 s    2 runs

Summary
  'hyper_threading_pr threads: 6' ran
    1.03 ± 0.05 times faster than 'hyper_threading_main threads: 6'

hyperfine -r 2 -n "hyper_threading_main threads: 8" 'RAYON_NUM_THREADS=8 ./hyper_threading_main' -n "hyper_threading_pr threads: 8" 'RAYON_NUM_THREADS=8 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 8
  Time (mean ± σ):     10.151 s ±  0.067 s    [User: 38.319 s, System: 0.921 s]
  Range (min … max):   10.104 s … 10.198 s    2 runs

Benchmark 2: hyper_threading_pr threads: 8
  Time (mean ± σ):     10.366 s ±  0.060 s    [User: 38.083 s, System: 0.969 s]
  Range (min … max):   10.324 s … 10.408 s    2 runs

Summary
  'hyper_threading_main threads: 8' ran
    1.02 ± 0.01 times faster than 'hyper_threading_pr threads: 8'

hyperfine -r 2 -n "hyper_threading_main threads: 16" 'RAYON_NUM_THREADS=16 ./hyper_threading_main' -n "hyper_threading_pr threads: 16" 'RAYON_NUM_THREADS=16 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 16
  Time (mean ± σ):     10.315 s ±  0.140 s    [User: 38.424 s, System: 1.013 s]
  Range (min … max):   10.216 s … 10.414 s    2 runs

Benchmark 2: hyper_threading_pr threads: 16
  Time (mean ± σ):     10.563 s ±  0.249 s    [User: 38.555 s, System: 0.952 s]
  Range (min … max):   10.387 s … 10.740 s    2 runs

Summary
  'hyper_threading_main threads: 16' ran
    1.02 ± 0.03 times faster than 'hyper_threading_pr threads: 16'
codecov[bot] commented 1 month ago

Codecov Report

Attention: Patch coverage is 20.68966% with 23 lines in your changes are missing coverage. Please review.

Project coverage is 94.76%. Comparing base (a896797) to head (17ca35b).

Files Patch % Lines
vm/src/serde/deserialize_program.rs 0.00% 15 Missing :warning:
vm/src/types/relocatable.rs 0.00% 2 Missing :warning:
vm/src/cairo_run.rs 0.00% 1 Missing :warning:
vm/src/hint_processor/hint_processor_definition.rs 0.00% 1 Missing :warning:
vm/src/types/builtin_name.rs 0.00% 1 Missing :warning:
vm/src/types/instruction.rs 0.00% 1 Missing :warning:
vm/src/types/layout_name.rs 0.00% 1 Missing :warning:
vm/src/types/program.rs 0.00% 1 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1755 +/- ## ======================================= Coverage 94.76% 94.76% ======================================= Files 101 101 Lines 38804 38804 ======================================= Hits 36773 36773 Misses 2031 2031 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

github-actions[bot] commented 1 month ago

Benchmark Results for unmodified programs :rocket:

Command Mean [s] Min [s] Max [s] Relative
base big_factorial 2.107 ± 0.126 2.032 2.441 1.03 ± 0.06
head big_factorial 2.041 ± 0.012 2.028 2.061 1.00
Command Mean [s] Min [s] Max [s] Relative
base big_fibonacci 2.004 ± 0.048 1.967 2.125 1.00 ± 0.03
head big_fibonacci 1.996 ± 0.033 1.977 2.084 1.00
Command Mean [s] Min [s] Max [s] Relative
base blake2s_integration_benchmark 7.639 ± 0.150 7.461 7.915 1.01 ± 0.02
head blake2s_integration_benchmark 7.562 ± 0.067 7.452 7.662 1.00
Command Mean [s] Min [s] Max [s] Relative
base compare_arrays_200000 2.118 ± 0.040 2.081 2.204 1.01 ± 0.02
head compare_arrays_200000 2.101 ± 0.018 2.080 2.142 1.00
Command Mean [s] Min [s] Max [s] Relative
base dict_integration_benchmark 1.429 ± 0.008 1.417 1.440 1.01 ± 0.01
head dict_integration_benchmark 1.420 ± 0.014 1.409 1.458 1.00
Command Mean [s] Min [s] Max [s] Relative
base field_arithmetic_get_square_benchmark 1.296 ± 0.020 1.267 1.322 1.01 ± 0.02
head field_arithmetic_get_square_benchmark 1.287 ± 0.012 1.275 1.309 1.00
Command Mean [s] Min [s] Max [s] Relative
base integration_builtins 7.777 ± 0.303 7.498 8.309 1.02 ± 0.04
head integration_builtins 7.614 ± 0.085 7.517 7.751 1.00
Command Mean [s] Min [s] Max [s] Relative
base keccak_integration_benchmark 8.149 ± 0.401 7.692 8.837 1.03 ± 0.05
head keccak_integration_benchmark 7.893 ± 0.150 7.737 8.289 1.00
Command Mean [s] Min [s] Max [s] Relative
base linear_search 2.095 ± 0.026 2.066 2.137 1.01 ± 0.01
head linear_search 2.069 ± 0.011 2.056 2.086 1.00
Command Mean [s] Min [s] Max [s] Relative
base math_cmp_and_pow_integration_benchmark 1.679 ± 0.009 1.670 1.695 1.00
head math_cmp_and_pow_integration_benchmark 1.680 ± 0.008 1.666 1.696 1.00 ± 0.01
Command Mean [s] Min [s] Max [s] Relative
base math_integration_benchmark 1.592 ± 0.006 1.584 1.601 1.00
head math_integration_benchmark 1.604 ± 0.026 1.588 1.674 1.01 ± 0.02
Command Mean [s] Min [s] Max [s] Relative
base memory_integration_benchmark 1.191 ± 0.006 1.184 1.202 1.00 ± 0.01
head memory_integration_benchmark 1.187 ± 0.005 1.180 1.195 1.00
Command Mean [s] Min [s] Max [s] Relative
base operations_with_data_structures_benchmarks 1.818 ± 0.027 1.789 1.886 1.00 ± 0.02
head operations_with_data_structures_benchmarks 1.817 ± 0.018 1.803 1.847 1.00
Command Mean [ms] Min [ms] Max [ms] Relative
base pedersen 513.2 ± 3.1 510.2 519.7 1.00
head pedersen 517.1 ± 9.2 510.6 537.9 1.01 ± 0.02
Command Mean [ms] Min [ms] Max [ms] Relative
base poseidon_integration_benchmark 963.6 ± 4.8 958.9 975.4 1.00 ± 0.01
head poseidon_integration_benchmark 962.0 ± 1.0 960.4 964.2 1.00
Command Mean [s] Min [s] Max [s] Relative
base secp_integration_benchmark 1.870 ± 0.015 1.852 1.897 1.01 ± 0.01
head secp_integration_benchmark 1.855 ± 0.008 1.843 1.865 1.00
Command Mean [ms] Min [ms] Max [ms] Relative
base set_integration_benchmark 647.5 ± 6.8 640.3 659.5 1.01 ± 0.01
head set_integration_benchmark 642.5 ± 2.3 639.6 647.0 1.00
Command Mean [s] Min [s] Max [s] Relative
base uint256_integration_benchmark 4.213 ± 0.134 4.136 4.587 1.00
head uint256_integration_benchmark 4.217 ± 0.065 4.129 4.327 1.00 ± 0.04