lambdaclass / cairo-vm

cairo-vm is a Rust implementation of the Cairo VM. Cairo (CPU Algebraic Intermediate Representation) is a programming language for writing provable programs, where one party can prove to another that a certain computation was executed correctly without the need for this party to re-execute the same program.
https://lambdaclass.github.io/cairo-vm
Apache License 2.0
514 stars 144 forks source link

Add `--run_from_cairo_pie` to `cairo-vm-cli` + workflow #1730

Closed fmoletta closed 5 months ago

fmoletta commented 5 months ago

Depends on #1729 & #1720 Adds --run_from_cairo_pie to cairo-vm-cli Adds a workflow that runs all test programs's pies and compares the trace, memory and pie outputs against cairo_lang

github-actions[bot] commented 5 months ago
**Hyper Thereading Benchmark results**

hyperfine -r 2 -n "hyper_threading_main threads: 1" 'RAYON_NUM_THREADS=1 ./hyper_threading_main' -n "hyper_threading_pr threads: 1" 'RAYON_NUM_THREADS=1 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 1
  Time (mean ± σ):     26.086 s ±  0.127 s    [User: 25.410 s, System: 0.673 s]
  Range (min … max):   25.996 s … 26.176 s    2 runs

Benchmark 2: hyper_threading_pr threads: 1
  Time (mean ± σ):     26.035 s ±  0.056 s    [User: 25.355 s, System: 0.677 s]
  Range (min … max):   25.996 s … 26.074 s    2 runs

Summary
  'hyper_threading_pr threads: 1' ran
    1.00 ± 0.01 times faster than 'hyper_threading_main threads: 1'

hyperfine -r 2 -n "hyper_threading_main threads: 2" 'RAYON_NUM_THREADS=2 ./hyper_threading_main' -n "hyper_threading_pr threads: 2" 'RAYON_NUM_THREADS=2 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 2
  Time (mean ± σ):     13.799 s ±  0.058 s    [User: 25.665 s, System: 0.720 s]
  Range (min … max):   13.757 s … 13.840 s    2 runs

Benchmark 2: hyper_threading_pr threads: 2
  Time (mean ± σ):     13.801 s ±  0.008 s    [User: 25.580 s, System: 0.756 s]
  Range (min … max):   13.795 s … 13.807 s    2 runs

Summary
  'hyper_threading_main threads: 2' ran
    1.00 ± 0.00 times faster than 'hyper_threading_pr threads: 2'

hyperfine -r 2 -n "hyper_threading_main threads: 4" 'RAYON_NUM_THREADS=4 ./hyper_threading_main' -n "hyper_threading_pr threads: 4" 'RAYON_NUM_THREADS=4 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 4
  Time (mean ± σ):     10.574 s ±  0.010 s    [User: 38.631 s, System: 0.853 s]
  Range (min … max):   10.567 s … 10.581 s    2 runs

Benchmark 2: hyper_threading_pr threads: 4
  Time (mean ± σ):     10.529 s ±  0.075 s    [User: 38.580 s, System: 0.897 s]
  Range (min … max):   10.476 s … 10.582 s    2 runs

Summary
  'hyper_threading_pr threads: 4' ran
    1.00 ± 0.01 times faster than 'hyper_threading_main threads: 4'

hyperfine -r 2 -n "hyper_threading_main threads: 6" 'RAYON_NUM_THREADS=6 ./hyper_threading_main' -n "hyper_threading_pr threads: 6" 'RAYON_NUM_THREADS=6 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 6
  Time (mean ± σ):     10.592 s ±  0.227 s    [User: 38.668 s, System: 0.856 s]
  Range (min … max):   10.431 s … 10.752 s    2 runs

Benchmark 2: hyper_threading_pr threads: 6
  Time (mean ± σ):     10.619 s ±  0.203 s    [User: 38.835 s, System: 0.966 s]
  Range (min … max):   10.475 s … 10.762 s    2 runs

Summary
  'hyper_threading_main threads: 6' ran
    1.00 ± 0.03 times faster than 'hyper_threading_pr threads: 6'

hyperfine -r 2 -n "hyper_threading_main threads: 8" 'RAYON_NUM_THREADS=8 ./hyper_threading_main' -n "hyper_threading_pr threads: 8" 'RAYON_NUM_THREADS=8 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 8
  Time (mean ± σ):     10.567 s ±  0.044 s    [User: 38.750 s, System: 1.006 s]
  Range (min … max):   10.536 s … 10.599 s    2 runs

Benchmark 2: hyper_threading_pr threads: 8
  Time (mean ± σ):     10.383 s ±  0.064 s    [User: 39.141 s, System: 0.911 s]
  Range (min … max):   10.338 s … 10.428 s    2 runs

Summary
  'hyper_threading_pr threads: 8' ran
    1.02 ± 0.01 times faster than 'hyper_threading_main threads: 8'

hyperfine -r 2 -n "hyper_threading_main threads: 16" 'RAYON_NUM_THREADS=16 ./hyper_threading_main' -n "hyper_threading_pr threads: 16" 'RAYON_NUM_THREADS=16 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 16
  Time (mean ± σ):     10.563 s ±  0.205 s    [User: 39.398 s, System: 0.991 s]
  Range (min … max):   10.418 s … 10.707 s    2 runs

Benchmark 2: hyper_threading_pr threads: 16
  Time (mean ± σ):     10.518 s ±  0.130 s    [User: 39.375 s, System: 0.971 s]
  Range (min … max):   10.426 s … 10.610 s    2 runs

Summary
  'hyper_threading_pr threads: 16' ran
    1.00 ± 0.02 times faster than 'hyper_threading_main threads: 16'
codecov[bot] commented 5 months ago

Codecov Report

Attention: Patch coverage is 56.25000% with 7 lines in your changes are missing coverage. Please review.

Project coverage is 94.79%. Comparing base (c5839fd) to head (c9bbbf9).

Files Patch % Lines
cairo-vm-cli/src/main.rs 56.25% 7 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1730 +/- ## ========================================== - Coverage 94.80% 94.79% -0.02% ========================================== Files 101 101 Lines 38719 38727 +8 ========================================== + Hits 36709 36710 +1 - Misses 2010 2017 +7 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

github-actions[bot] commented 5 months ago

Benchmark Results for unmodified programs :rocket:

Command Mean [s] Min [s] Max [s] Relative
base big_factorial 2.074 ± 0.025 2.040 2.110 1.00 ± 0.01
head big_factorial 2.073 ± 0.013 2.055 2.099 1.00
Command Mean [s] Min [s] Max [s] Relative
base big_fibonacci 2.088 ± 0.034 2.048 2.153 1.02 ± 0.02
head big_fibonacci 2.053 ± 0.030 2.018 2.109 1.00
Command Mean [s] Min [s] Max [s] Relative
base blake2s_integration_benchmark 7.070 ± 0.091 6.940 7.207 1.00
head blake2s_integration_benchmark 7.084 ± 0.068 6.998 7.175 1.00 ± 0.02
Command Mean [s] Min [s] Max [s] Relative
base compare_arrays_200000 2.130 ± 0.027 2.098 2.180 1.01 ± 0.01
head compare_arrays_200000 2.115 ± 0.013 2.099 2.142 1.00
Command Mean [s] Min [s] Max [s] Relative
base dict_integration_benchmark 1.402 ± 0.025 1.375 1.459 1.01 ± 0.02
head dict_integration_benchmark 1.391 ± 0.018 1.371 1.433 1.00
Command Mean [s] Min [s] Max [s] Relative
base field_arithmetic_get_square_benchmark 1.237 ± 0.015 1.223 1.274 1.00
head field_arithmetic_get_square_benchmark 1.257 ± 0.030 1.228 1.318 1.02 ± 0.03
Command Mean [s] Min [s] Max [s] Relative
base integration_builtins 7.119 ± 0.091 7.009 7.240 1.00
head integration_builtins 7.143 ± 0.076 7.066 7.304 1.00 ± 0.02
Command Mean [s] Min [s] Max [s] Relative
base keccak_integration_benchmark 7.288 ± 0.117 7.171 7.556 1.00
head keccak_integration_benchmark 7.298 ± 0.103 7.235 7.567 1.00 ± 0.02
Command Mean [s] Min [s] Max [s] Relative
base linear_search 2.163 ± 0.033 2.132 2.231 1.00
head linear_search 2.180 ± 0.052 2.152 2.320 1.01 ± 0.03
Command Mean [s] Min [s] Max [s] Relative
base math_cmp_and_pow_integration_benchmark 1.784 ± 0.013 1.766 1.804 1.00
head math_cmp_and_pow_integration_benchmark 1.806 ± 0.058 1.764 1.934 1.01 ± 0.03
Command Mean [s] Min [s] Max [s] Relative
base math_integration_benchmark 1.582 ± 0.015 1.566 1.610 1.00 ± 0.01
head math_integration_benchmark 1.576 ± 0.011 1.563 1.599 1.00
Command Mean [s] Min [s] Max [s] Relative
base memory_integration_benchmark 1.225 ± 0.023 1.200 1.275 1.01 ± 0.02
head memory_integration_benchmark 1.211 ± 0.008 1.200 1.226 1.00
Command Mean [s] Min [s] Max [s] Relative
base operations_with_data_structures_benchmarks 1.855 ± 0.016 1.831 1.886 1.01 ± 0.01
head operations_with_data_structures_benchmarks 1.844 ± 0.013 1.832 1.873 1.00
Command Mean [ms] Min [ms] Max [ms] Relative
base pedersen 531.9 ± 5.8 527.5 548.0 1.01 ± 0.01
head pedersen 526.8 ± 1.1 525.2 528.4 1.00
Command Mean [ms] Min [ms] Max [ms] Relative
base poseidon_integration_benchmark 972.0 ± 9.5 962.6 987.6 1.00
head poseidon_integration_benchmark 975.1 ± 12.5 963.7 1007.9 1.00 ± 0.02
Command Mean [s] Min [s] Max [s] Relative
base secp_integration_benchmark 1.857 ± 0.012 1.841 1.879 1.01 ± 0.01
head secp_integration_benchmark 1.847 ± 0.008 1.839 1.861 1.00
Command Mean [ms] Min [ms] Max [ms] Relative
base set_integration_benchmark 694.3 ± 5.1 688.6 703.5 1.00
head set_integration_benchmark 728.6 ± 5.6 724.0 743.8 1.05 ± 0.01
Command Mean [s] Min [s] Max [s] Relative
base uint256_integration_benchmark 4.054 ± 0.027 4.012 4.086 1.00
head uint256_integration_benchmark 4.058 ± 0.016 4.041 4.090 1.00 ± 0.01