search

privacy-scaling-explorations / zkevm-circuits

https://privacy-scaling-explorations.github.io/zkevm-circuits/
Other
815 stars 853 forks source link

Integration test failed #1439

Open wangtsiao opened 1 year ago

wangtsiao commented 1 year ago

What command(s) is the bug in?

cargo test --release --test circuits real_prover::serial_test_super_circuit_multiple_erc20_openzeppelin_transfers -- --nocapture 2>&1

Describe the bug

mock_prover passes tests, but the following error occurs when switching to real_prover.

Not only circuit_multiple_erc20_openzeppelin_transfers, but also circuit_block_transfer_0, circuit_multiple_transfers_0, and circuit_erc20_openzeppelin_transfer_succeed.

cargo test --release --test circuits real_prover::serial_test_super_circuit_multiple_erc20_openzeppelin_transfers -- --nocapture 2>&1
    Finished release [optimized] target(s) in 0.77s
     Running tests/circuits.rs (/home/root/zkevm-circuits/target/release/deps/circuits-7a8fd0c23e232ec8)

running 1 test
[2023-05-30T07:34:55Z INFO  integration_tests::integration_test_circuits] test Super circuit, block: #8 - Multiple ERC20 OpenZeppelin transfers
test real_prover::serial_test_super_circuit_multiple_erc20_openzeppelin_transfers has been running for over 60 seconds
thread 'real_prover::serial_test_super_circuit_multiple_erc20_openzeppelin_transfers' panicked at 'proof generation should not fail: ConstraintSystemFailure', /home/root/zkevm-circuits/integration-tests/src/integration_test_circuits.rs:192:14
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
test real_prover::serial_test_super_circuit_multiple_erc20_openzeppelin_transfers ... FAILED

failures:

failures:
    real_prover::serial_test_super_circuit_multiple_erc20_openzeppelin_transfers

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 79 filtered out; finished in 1182.33s

error: test failed, to rerun pass `--test circuits`

Concrete steps to reproduce the bug. If it's able reproduce via testool, please share test_id from jenkins report

cd integration-tests
cargo test --release --test circuits real_prover::serial_test_super_circuit_multiple_erc20_openzeppelin_transfers -- --nocapture 2>&1
ed255 commented 1 year ago

Thanks for the report, we're currently investigating this.

I have run the integration tests for each individual circuit with the real prover and they all seem to pass.

I used this script:

CIRCUITS="bytecode_circuit
copy_circuit
evm_circuit
exp_circuit
keccak_circuit
state_circuit
tx_circuit"
# super_circuit"

TESTS="block_transfer_0
multiple_transfers_0
erc20_openzeppelin_transfer_succeed
erc20_openzeppelin_transfer_fail
multiple_erc20_openzeppelin_transfers"

for circuit in $CIRCUITS; do
    for test in $TESTS; do
        cargo test --release --test circuits real_prover::serial_test_${circuit}_${test} -- --nocapture
    done
done

This are the results:

Click to expand ``` running 1 test test real_prover::serial_test_bytecode_circuit_block_transfer_0 ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 9.45s running 1 test test real_prover::serial_test_bytecode_circuit_multiple_transfers_0 ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 9.30s running 1 test test real_prover::serial_test_bytecode_circuit_erc20_openzeppelin_transfer_succeed ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 9.64s running 1 test test real_prover::serial_test_bytecode_circuit_erc20_openzeppelin_transfer_fail ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 9.64s running 1 test test real_prover::serial_test_bytecode_circuit_multiple_erc20_openzeppelin_transfers ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 9.62s running 1 test test real_prover::serial_test_copy_circuit_block_transfer_0 ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 15.26s running 1 test test real_prover::serial_test_copy_circuit_multiple_transfers_0 ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 15.16s running 1 test test real_prover::serial_test_copy_circuit_erc20_openzeppelin_transfer_succeed ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 15.51s running 1 test test real_prover::serial_test_copy_circuit_erc20_openzeppelin_transfer_fail ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 15.44s running 1 test test real_prover::serial_test_copy_circuit_multiple_erc20_openzeppelin_transfers ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 15.57s running 1 test test real_prover::serial_test_evm_circuit_block_transfer_0 has been running for over 60 seconds test real_prover::serial_test_evm_circuit_block_transfer_0 ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 461.67s running 1 test test real_prover::serial_test_evm_circuit_multiple_transfers_0 has been running for over 60 seconds test real_prover::serial_test_evm_circuit_multiple_transfers_0 ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 462.29s running 1 test test real_prover::serial_test_evm_circuit_erc20_openzeppelin_transfer_succeed has been running for over 60 seconds test real_prover::serial_test_evm_circuit_erc20_openzeppelin_transfer_succeed ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 462.89s running 1 test test real_prover::serial_test_evm_circuit_erc20_openzeppelin_transfer_fail has been running for over 60 seconds test real_prover::serial_test_evm_circuit_erc20_openzeppelin_transfer_fail ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 466.73s running 1 test test real_prover::serial_test_evm_circuit_multiple_erc20_openzeppelin_transfers has been running for over 60 seconds test real_prover::serial_test_evm_circuit_multiple_erc20_openzeppelin_transfers ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 463.89s running 1 test test real_prover::serial_test_exp_circuit_block_transfer_0 ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 5.38s running 1 test test real_prover::serial_test_exp_circuit_multiple_transfers_0 ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 5.39s running 1 test test real_prover::serial_test_exp_circuit_erc20_openzeppelin_transfer_succeed ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 5.38s running 1 test test real_prover::serial_test_exp_circuit_erc20_openzeppelin_transfer_fail ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 5.34s running 1 test test real_prover::serial_test_exp_circuit_multiple_erc20_openzeppelin_transfers ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 5.42s running 1 test test real_prover::serial_test_keccak_circuit_block_transfer_0 has been running for over 60 seconds test real_prover::serial_test_keccak_circuit_block_transfer_0 ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 76.31s running 1 test test real_prover::serial_test_keccak_circuit_multiple_transfers_0 has been running for over 60 seconds test real_prover::serial_test_keccak_circuit_multiple_transfers_0 ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 77.55s running 1 test test real_prover::serial_test_keccak_circuit_erc20_openzeppelin_transfer_succeed has been running for over 60 seconds test real_prover::serial_test_keccak_circuit_erc20_openzeppelin_transfer_succeed ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 95.78s running 1 test test real_prover::serial_test_keccak_circuit_erc20_openzeppelin_transfer_fail has been running for over 60 seconds test real_prover::serial_test_keccak_circuit_erc20_openzeppelin_transfer_fail ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 92.32s running 1 test test real_prover::serial_test_keccak_circuit_multiple_erc20_openzeppelin_transfers has been running for over 60 seconds test real_prover::serial_test_keccak_circuit_multiple_erc20_openzeppelin_transfers ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 97.02s running 1 test test real_prover::serial_test_state_circuit_block_transfer_0 has been running for over 60 seconds test real_prover::serial_test_state_circuit_block_transfer_0 ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 73.43s running 1 test test real_prover::serial_test_state_circuit_multiple_transfers_0 has been running for over 60 seconds test real_prover::serial_test_state_circuit_multiple_transfers_0 ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 73.68s running 1 test test real_prover::serial_test_state_circuit_erc20_openzeppelin_transfer_succeed has been running for over 60 seconds test real_prover::serial_test_state_circuit_erc20_openzeppelin_transfer_succeed ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 79.10s running 1 test test real_prover::serial_test_state_circuit_erc20_openzeppelin_transfer_fail has been running for over 60 seconds test real_prover::serial_test_state_circuit_erc20_openzeppelin_transfer_fail ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 79.03s running 1 test test real_prover::serial_test_state_circuit_multiple_erc20_openzeppelin_transfers has been running for over 60 seconds test real_prover::serial_test_state_circuit_multiple_erc20_openzeppelin_transfers ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 79.19s running 1 test test real_prover::serial_test_tx_circuit_block_transfer_0 has been running for over 60 seconds test real_prover::serial_test_tx_circuit_block_transfer_0 ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 221.28s running 1 test test real_prover::serial_test_tx_circuit_multiple_transfers_0 has been running for over 60 seconds test real_prover::serial_test_tx_circuit_multiple_transfers_0 ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 222.70s running 1 test test real_prover::serial_test_tx_circuit_erc20_openzeppelin_transfer_succeed has been running for over 60 seconds test real_prover::serial_test_tx_circuit_erc20_openzeppelin_transfer_succeed ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 220.84s running 1 test test real_prover::serial_test_tx_circuit_erc20_openzeppelin_transfer_fail has been running for over 60 seconds test real_prover::serial_test_tx_circuit_erc20_openzeppelin_transfer_fail ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 221.93s running 1 test test real_prover::serial_test_tx_circuit_multiple_erc20_openzeppelin_transfers has been running for over 60 seconds test real_prover::serial_test_tx_circuit_multiple_erc20_openzeppelin_transfers ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 79 filtered out; finished in 221.62s ```
ed255 commented 1 year ago

Idea: we can compare the status of this issue with the scroll fork (to see if we're failing the same tests or not)

AronisAt79 commented 1 year ago

Thanks @ed255 for the analysis. integration tests in e2e environment are for the most part successful, apart from the EVM and SUPER circuit proofs, Reference issues : #1334 #1335 (all tests run with aggregate enabled) Super circuit test without aggregate is currently in progress. Will update here as soon as test concludes.

ed255 commented 1 year ago

Here's a summary of what we have tested so far:

Results for real prover with:

Circuit standalone with Aggregation
EVM PASS FAILED
State PASS PASS
Tx PASS PASS
Bytecode PASS PASS
Copy PASS PASS
Keccak PASS PASS
Exp PASS PASS
Pi PASS PASS
Super FAILED FAILED
AronisAt79 commented 1 year ago

proof for super circuit / no aggregate succeeds. https://zkevm-chain-testing.s3.eu-central-1.amazonaws.com/proveSingeCrossChainTx_super_False_410-maronis.tar.gz

ed255 commented 1 year ago

We have been able to reproduce this issue. All integration tests with the real prover and super circuit are failing https://github.com/privacy-scaling-explorations/zkevm-circuits/actions/runs/5152993138/jobs/9279691434

The fact that the subcircuits test with real prover pass, and that all mock prover tests pass makes it a bit hard to debug this issue. A ConstraintSystemFailure is very opaque and doesn't give you any hint of what may be the problem (which ideally should be found by the MockProver, but that's not happening)

ed255 commented 1 year ago

proof for super circuit / no aggregate succeeds. https://zkevm-chain-testing.s3.eu-central-1.amazonaws.com/proveSingeCrossChainTx_super_False_410-maronis.tar.gz

Update on this: The test that succeeded was using an old version of the zkevm-circuits. Via the integration test run via github actions we have been able to reproduce the issue and observe the failure (see previous message)