search

lambdaclass / cairo-vm

cairo-vm is a Rust implementation of the Cairo VM. Cairo (CPU Algebraic Intermediate Representation) is a programming language for writing provable programs, where one party can prove to another that a certain computation was executed correctly without the need for this party to re-execute the same program.
https://lambdaclass.github.io/cairo-vm
Apache License 2.0
522 stars 150 forks source link

Add Debug scripts #1839

Closed JulianGCalderon closed 1 month ago

JulianGCalderon commented 2 months ago

Add debug scripts

This PR was initially inside of #1824, but I moved it to reduce the size of that PR.

Description

It adds some scripts (and documents them) for debugging differences in memory, trace, public and private air inputs.

Checklist

github-actions[bot] commented 2 months ago 
**Hyper Thereading Benchmark results**

hyperfine -r 2 -n "hyper_threading_main threads: 1" 'RAYON_NUM_THREADS=1 ./hyper_threading_main' -n "hyper_threading_pr threads: 1" 'RAYON_NUM_THREADS=1 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 1
  Time (mean ± σ):     30.108 s ±  0.065 s    [User: 29.330 s, System: 0.776 s]
  Range (min … max):   30.062 s … 30.155 s    2 runs

Benchmark 2: hyper_threading_pr threads: 1
  Time (mean ± σ):     30.469 s ±  0.130 s    [User: 29.685 s, System: 0.782 s]
  Range (min … max):   30.377 s … 30.561 s    2 runs

Summary
  'hyper_threading_main threads: 1' ran
    1.01 ± 0.00 times faster than 'hyper_threading_pr threads: 1'

hyperfine -r 2 -n "hyper_threading_main threads: 2" 'RAYON_NUM_THREADS=2 ./hyper_threading_main' -n "hyper_threading_pr threads: 2" 'RAYON_NUM_THREADS=2 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 2
  Time (mean ± σ):     16.780 s ±  0.023 s    [User: 29.555 s, System: 0.776 s]
  Range (min … max):   16.763 s … 16.796 s    2 runs

Benchmark 2: hyper_threading_pr threads: 2
  Time (mean ± σ):     16.912 s ±  0.075 s    [User: 29.881 s, System: 0.763 s]
  Range (min … max):   16.859 s … 16.966 s    2 runs

Summary
  'hyper_threading_main threads: 2' ran
    1.01 ± 0.00 times faster than 'hyper_threading_pr threads: 2'

hyperfine -r 2 -n "hyper_threading_main threads: 4" 'RAYON_NUM_THREADS=4 ./hyper_threading_main' -n "hyper_threading_pr threads: 4" 'RAYON_NUM_THREADS=4 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 4
  Time (mean ± σ):     11.444 s ±  0.184 s    [User: 41.568 s, System: 0.977 s]
  Range (min … max):   11.313 s … 11.574 s    2 runs

Benchmark 2: hyper_threading_pr threads: 4
  Time (mean ± σ):     11.502 s ±  0.688 s    [User: 41.806 s, System: 0.968 s]
  Range (min … max):   11.015 s … 11.988 s    2 runs

Summary
  'hyper_threading_main threads: 4' ran
    1.01 ± 0.06 times faster than 'hyper_threading_pr threads: 4'

hyperfine -r 2 -n "hyper_threading_main threads: 6" 'RAYON_NUM_THREADS=6 ./hyper_threading_main' -n "hyper_threading_pr threads: 6" 'RAYON_NUM_THREADS=6 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 6
  Time (mean ± σ):     11.369 s ±  0.026 s    [User: 41.853 s, System: 0.951 s]
  Range (min … max):   11.351 s … 11.387 s    2 runs

Benchmark 2: hyper_threading_pr threads: 6
  Time (mean ± σ):     11.513 s ±  0.475 s    [User: 41.796 s, System: 0.999 s]
  Range (min … max):   11.176 s … 11.849 s    2 runs

Summary
  'hyper_threading_main threads: 6' ran
    1.01 ± 0.04 times faster than 'hyper_threading_pr threads: 6'

hyperfine -r 2 -n "hyper_threading_main threads: 8" 'RAYON_NUM_THREADS=8 ./hyper_threading_main' -n "hyper_threading_pr threads: 8" 'RAYON_NUM_THREADS=8 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 8
  Time (mean ± σ):     11.237 s ±  0.159 s    [User: 41.822 s, System: 0.979 s]
  Range (min … max):   11.125 s … 11.349 s    2 runs

Benchmark 2: hyper_threading_pr threads: 8
  Time (mean ± σ):     11.364 s ±  0.032 s    [User: 41.818 s, System: 1.002 s]
  Range (min … max):   11.341 s … 11.387 s    2 runs

Summary
  'hyper_threading_main threads: 8' ran
    1.01 ± 0.01 times faster than 'hyper_threading_pr threads: 8'

hyperfine -r 2 -n "hyper_threading_main threads: 16" 'RAYON_NUM_THREADS=16 ./hyper_threading_main' -n "hyper_threading_pr threads: 16" 'RAYON_NUM_THREADS=16 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 16
  Time (mean ± σ):     11.173 s ±  0.198 s    [User: 42.185 s, System: 1.092 s]
  Range (min … max):   11.033 s … 11.313 s    2 runs

Benchmark 2: hyper_threading_pr threads: 16
  Time (mean ± σ):     11.411 s ±  0.043 s    [User: 42.031 s, System: 1.083 s]
  Range (min … max):   11.381 s … 11.442 s    2 runs

Summary
  'hyper_threading_main threads: 16' ran
    1.02 ± 0.02 times faster than 'hyper_threading_pr threads: 16'
github-actions[bot] commented 2 months ago

Benchmark Results for unmodified programs :rocket:

Command Mean [s] Min [s] Max [s] Relative
base big_factorial 2.392 ± 0.014 2.376 2.416 1.00
head big_factorial 2.401 ± 0.012 2.382 2.429 1.00 ± 0.01
Command Mean [s] Min [s] Max [s] Relative
base big_fibonacci 2.361 ± 0.062 2.313 2.515 1.01 ± 0.03
head big_fibonacci 2.332 ± 0.021 2.310 2.384 1.00
Command Mean [s] Min [s] Max [s] Relative
base blake2s_integration_benchmark 8.626 ± 0.190 8.436 8.934 1.00
head blake2s_integration_benchmark 8.810 ± 0.322 8.482 9.491 1.02 ± 0.04
Command Mean [s] Min [s] Max [s] Relative
base compare_arrays_200000 2.525 ± 0.063 2.473 2.669 1.00 ± 0.04
head compare_arrays_200000 2.524 ± 0.066 2.472 2.686 1.00
Command Mean [s] Min [s] Max [s] Relative
base dict_integration_benchmark 1.652 ± 0.006 1.644 1.661 1.01 ± 0.01
head dict_integration_benchmark 1.634 ± 0.008 1.628 1.653 1.00
Command Mean [s] Min [s] Max [s] Relative
base field_arithmetic_get_square_benchmark 1.396 ± 0.036 1.366 1.469 1.02 ± 0.03
head field_arithmetic_get_square_benchmark 1.368 ± 0.010 1.353 1.388 1.00
Command Mean [s] Min [s] Max [s] Relative
base integration_builtins 8.932 ± 0.501 8.548 10.289 1.01 ± 0.07
head integration_builtins 8.831 ± 0.287 8.522 9.515 1.00
Command Mean [s] Min [s] Max [s] Relative
base keccak_integration_benchmark 9.065 ± 0.206 8.810 9.470 1.00
head keccak_integration_benchmark 9.126 ± 0.381 8.771 10.131 1.01 ± 0.05
Command Mean [s] Min [s] Max [s] Relative
base linear_search 2.500 ± 0.041 2.465 2.590 1.00
head linear_search 2.527 ± 0.077 2.463 2.704 1.01 ± 0.04
Command Mean [s] Min [s] Max [s] Relative
base math_cmp_and_pow_integration_benchmark 1.713 ± 0.020 1.696 1.767 1.00
head math_cmp_and_pow_integration_benchmark 1.730 ± 0.061 1.697 1.866 1.01 ± 0.04
Command Mean [s] Min [s] Max [s] Relative
base math_integration_benchmark 1.671 ± 0.019 1.657 1.725 1.01 ± 0.01
head math_integration_benchmark 1.661 ± 0.015 1.647 1.689 1.00
Command Mean [s] Min [s] Max [s] Relative
base memory_integration_benchmark 1.400 ± 0.014 1.389 1.436 1.00
head memory_integration_benchmark 1.402 ± 0.013 1.388 1.429 1.00 ± 0.01
Command Mean [s] Min [s] Max [s] Relative
base operations_with_data_structures_benchmarks 1.780 ± 0.030 1.760 1.860 1.01 ± 0.02
head operations_with_data_structures_benchmarks 1.769 ± 0.012 1.748 1.793 1.00
Command Mean [ms] Min [ms] Max [ms] Relative
base pedersen 571.3 ± 2.6 566.9 576.1 1.00 ± 0.00
head pedersen 570.8 ± 1.2 569.3 573.0 1.00
Command Mean [ms] Min [ms] Max [ms] Relative
base poseidon_integration_benchmark 674.9 ± 6.7 665.4 685.9 1.00 ± 0.01
head poseidon_integration_benchmark 673.6 ± 5.9 665.2 684.6 1.00
Command Mean [s] Min [s] Max [s] Relative
base secp_integration_benchmark 1.984 ± 0.014 1.968 2.018 1.00
head secp_integration_benchmark 1.999 ± 0.036 1.973 2.095 1.01 ± 0.02
Command Mean [ms] Min [ms] Max [ms] Relative
base set_integration_benchmark 683.6 ± 5.8 677.8 693.2 1.00 ± 0.02
head set_integration_benchmark 683.1 ± 11.6 676.2 714.6 1.00
Command Mean [s] Min [s] Max [s] Relative
base uint256_integration_benchmark 4.808 ± 0.112 4.698 5.076 1.00
head uint256_integration_benchmark 4.818 ± 0.108 4.716 5.081 1.00 ± 0.03
codecov[bot] commented 2 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 94.83%. Comparing base (5f8a011) to head (6f292f3). Report is 1 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1839 +/- ## ======================================= Coverage 94.83% 94.83% ======================================= Files 101 101 Lines 39579 39579 ======================================= Hits 37536 37536 Misses 2043 2043 ``` | [Flag](https://app.codecov.io/gh/lambdaclass/cairo-vm/pull/1839/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lambdaclass) | Coverage Δ | | |---|---|---| | [](https://app.codecov.io/gh/lambdaclass/cairo-vm/pull/1839/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lambdaclass) | `94.83% <ø> (?)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lambdaclass#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.