MPT: Testing & go library fix

adria0 commented 8 months ago

Description

This PR:

Removes the light client (that now is deprecated) and moves its StateUpdate circuit into bin/mpt-test.
bin/mpt-test is a tool to prove mainnet tests, see the README inside for more information.
Is not possible to link to external golang libraries in the same binary, so this PR also moves golang MPT code inside /geth-utils

Type of change

[X] New feature (non-breaking change which adds functionality)

Removal if light-client-poc folder
Move mpt-witness-generator into geth-utils, this includes also renaming the git action workflow file to geth utils.
Create new bin/mpt-folder
- access-lists contains JSON files of accesslists of mainnet blocks
- src/circuit contains a circuit and the witness generator to test chained MPT proofs. Well, there's some stuff that is not strictly mandatory to be there - it's taken from light client circuit - but it can be done later.
- src/cache.rs it's a proxy server with a cache file. There's a lot of RPC calls to execute this tests and this allows to cache them all into a file, this makes the testing being possible without accessing any RPC server and speeding-up it.

How Has This Been Tested?

For the moment, some of the tests are not passing, but the idea is to integrate into CI when tests passes successfully.

lispc commented 8 months ago

one comment: "not possible to link to external golang libraries in the same binary". Not sure about this, in our repo we use dynamic library (.so), so we can put 2 golang libs inside 1 rust main bin.

adria0 commented 8 months ago

Nice! Only one comment: it would be good to have instructions how to run the tests (how to ignore keccak ...) in bin/mpt-test/README.md.

Keccak is ignored by default!

https://github.com/privacy-scaling-explorations/zkevm-circuits/blob/e3ffe4450c75dc5bac3dc4f09d8524337038f7b7/bin/mpt-test/Cargo.toml#L26

So probally it will be nice to explain HOW to run them with keccak 👍 I will add a comment on this.

The idea is to run the tests by using test_mainnet_blocks.sh that automatically downloads the caché file and runs the tests, as is explained in the README.md, do you think that needs to be somehow improved?

Edit: added 0f6d620

adria0 commented 8 months ago

one comment: "not possible to link to external golang libraries in the same binary". Not sure about this, in our repo we use dynamic library (.so), so we can put 2 golang libs inside 1 rust main bin.

If I understand correctly, you linked two .so without problems? Btw, is not an .rlib instead a .so?

Happened to @rrtoledo that when trying to integrate MPT circuit go library in bus-mapping the it crashed really badly with minpc= 0x105e6dc20 min= 0x105e6dc20 maxpc= 0x1064282c0 max= 0x1064283b0 fatal error: minpc or maxpc invalid runtime: panic before malloc heap initialized.

When we joined the two golang libraries everything worked ok, so we just glued together the libraries.

rrtoledo commented 8 months ago

The code looks good to me! Only problem is I cannot run the script test_mainnet_blocks. I have been looking at it since yesterday with Adria.

miha-stopar commented 8 months ago

Nice! Only one comment: it would be good to have instructions how to run the tests (how to ignore keccak ...) in bin/mpt-test/README.md.

Keccak is ignored by default!

https://github.com/privacy-scaling-explorations/zkevm-circuits/blob/e3ffe4450c75dc5bac3dc4f09d8524337038f7b7/bin/mpt-test/Cargo.toml#L26

So probally it will be nice to explain HOW to run them with keccak 👍 I will add a comment on this.

The idea is to run the tests by using test_mainnet_blocks.sh that automatically downloads the caché file and runs the tests, as is explained in the README.md, do you think that needs to be somehow improved?

Edit: added 0f6d620

Yep, sorry, meant with keccak. Thanks!

privacy-scaling-explorations / zkevm-circuits