lambdaclass / cairo-vm

cairo-vm is a Rust implementation of the Cairo VM. Cairo (CPU Algebraic Intermediate Representation) is a programming language for writing provable programs, where one party can prove to another that a certain computation was executed correctly without the need for this party to re-execute the same program.
https://lambdaclass.github.io/cairo-vm
Apache License 2.0
504 stars 138 forks source link

Add workflow to compare vm outputs for factorial on all layouts #1653

Closed fmoletta closed 6 months ago

fmoletta commented 6 months ago

Depends on #1651 Also Fixes:

codecov[bot] commented 6 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 97.68%. Comparing base (7566aa5) to head (0c727bb).

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1653 +/- ## ======================================= Coverage 97.68% 97.68% ======================================= Files 91 91 Lines 37668 37689 +21 ======================================= + Hits 36796 36817 +21 Misses 872 872 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

github-actions[bot] commented 6 months ago

Benchmark Results for unmodified programs :rocket:

Command Mean [s] Min [s] Max [s] Relative
base big_factorial 2.173 ± 0.025 2.133 2.215 1.00
head big_factorial 2.260 ± 0.053 2.198 2.364 1.04 ± 0.03
Command Mean [s] Min [s] Max [s] Relative
base big_fibonacci 2.238 ± 0.021 2.199 2.268 1.03 ± 0.02
head big_fibonacci 2.182 ± 0.045 2.132 2.284 1.00
Command Mean [s] Min [s] Max [s] Relative
base blake2s_integration_benchmark 8.192 ± 0.311 7.937 9.026 1.00
head blake2s_integration_benchmark 8.538 ± 0.268 8.205 9.205 1.04 ± 0.05
Command Mean [s] Min [s] Max [s] Relative
base compare_arrays_200000 2.258 ± 0.077 2.195 2.469 1.00
head compare_arrays_200000 2.311 ± 0.023 2.274 2.345 1.02 ± 0.04
Command Mean [s] Min [s] Max [s] Relative
base dict_integration_benchmark 1.398 ± 0.016 1.371 1.422 1.00
head dict_integration_benchmark 1.458 ± 0.014 1.438 1.478 1.04 ± 0.02
Command Mean [s] Min [s] Max [s] Relative
base field_arithmetic_get_square_benchmark 1.273 ± 0.040 1.219 1.373 1.00
head field_arithmetic_get_square_benchmark 1.286 ± 0.016 1.256 1.311 1.01 ± 0.03
Command Mean [s] Min [s] Max [s] Relative
base integration_builtins 8.188 ± 0.126 8.012 8.413 1.00
head integration_builtins 8.486 ± 0.137 8.295 8.751 1.04 ± 0.02
Command Mean [s] Min [s] Max [s] Relative
base keccak_integration_benchmark 8.430 ± 0.130 8.271 8.633 1.00
head keccak_integration_benchmark 8.633 ± 0.196 8.443 9.050 1.02 ± 0.03
Command Mean [s] Min [s] Max [s] Relative
base linear_search 2.244 ± 0.023 2.215 2.288 1.00
head linear_search 2.293 ± 0.027 2.254 2.338 1.02 ± 0.02
Command Mean [s] Min [s] Max [s] Relative
base math_cmp_and_pow_integration_benchmark 1.507 ± 0.013 1.488 1.523 1.00
head math_cmp_and_pow_integration_benchmark 1.556 ± 0.011 1.544 1.574 1.03 ± 0.01
Command Mean [s] Min [s] Max [s] Relative
base math_integration_benchmark 1.385 ± 0.036 1.356 1.474 1.00
head math_integration_benchmark 1.409 ± 0.015 1.390 1.433 1.02 ± 0.03
Command Mean [s] Min [s] Max [s] Relative
base memory_integration_benchmark 1.221 ± 0.035 1.181 1.299 1.00
head memory_integration_benchmark 1.253 ± 0.015 1.236 1.288 1.03 ± 0.03
Command Mean [s] Min [s] Max [s] Relative
base operations_with_data_structures_benchmarks 1.556 ± 0.017 1.525 1.586 1.00
head operations_with_data_structures_benchmarks 1.581 ± 0.016 1.561 1.608 1.02 ± 0.02
Command Mean [ms] Min [ms] Max [ms] Relative
base pedersen 576.1 ± 7.2 566.5 586.2 1.00
head pedersen 578.9 ± 5.5 568.4 585.1 1.00 ± 0.02
Command Mean [ms] Min [ms] Max [ms] Relative
base poseidon_integration_benchmark 949.9 ± 10.8 931.3 967.8 1.00
head poseidon_integration_benchmark 973.9 ± 13.2 955.7 996.7 1.03 ± 0.02
Command Mean [s] Min [s] Max [s] Relative
base secp_integration_benchmark 1.825 ± 0.025 1.790 1.872 1.00
head secp_integration_benchmark 1.870 ± 0.039 1.818 1.933 1.03 ± 0.03
Command Mean [ms] Min [ms] Max [ms] Relative
base set_integration_benchmark 717.7 ± 2.7 711.9 720.6 1.01 ± 0.02
head set_integration_benchmark 711.8 ± 11.1 691.0 724.3 1.00
Command Mean [s] Min [s] Max [s] Relative
base uint256_integration_benchmark 4.493 ± 0.079 4.394 4.630 1.00
head uint256_integration_benchmark 4.566 ± 0.055 4.504 4.695 1.02 ± 0.02