search

codex-storage / codex-contracts-eth

Ethereum smart contracts for Codex
Other
5 stars 9 forks source link

Use fuzzing to check invariants #37

Closed markspanbroek closed 1 year ago

markspanbroek commented 1 year ago

Simplifies the contract code by moving the invariant checks to separate files. Uses echidna to perform the fuzzing.

markspanbroek commented 1 year ago

Somehow locally fuzzing/corpus/coverage isn't producing the transactions needed to run the fuzzer.

No need to worry about the corpus; this is a list of transactions that the fuzzer tried before, which it uses to be more efficient in finding cases that it hasn't covered before. It starts out empty, which is what you see on your development machine. I've configured the CI to cache the results, so that it remembers its corpus over multiple runs.

The real reason that your run fails, seems to be this line:

echidna-test: Given contract "FuzzCollateral" not found in given file

Can you try to compile the contracts beforehand? For instance using npx hardhat compile? If that solves the problem then we probably need to include that step in fuzz.sh, just before running docker.

emizzle commented 1 year ago

Somehow locally fuzzing/corpus/coverage isn't producing the transactions needed to run the fuzzer.

No need to worry about the corpus; this is a list of transactions that the fuzzer tried before, which it uses to be more efficient in finding cases that it hasn't covered before. It starts out empty, which is what you see on your development machine. I've configured the CI to cache the results, so that it remembers its corpus over multiple runs.

The real reason that your run fails, seems to be this line:

echidna-test: Given contract "FuzzCollateral" not found in given file

Can you try to compile the contracts beforehand? For instance using npx hardhat compile? If that solves the problem then we probably need to include that step in fuzz.sh, just before running docker.

I did try compiling the contracts earlier via npm test, however the artifacts may have been deleted after the run, so I compiled via npx hardhat compile then npm run fuzz, and got a "blank" error:

❯ npx hardhat compile
Compiled 6 Solidity files successfully
❯ npm run fuzz

> fuzz
> fuzzing/fuzz.sh

Loaded total of 0 transactions from fuzzing/corpus/reproducers/
Loaded total of 0 transactions from fuzzing/corpus/coverage/
Analyzing contract: /src/contracts/FuzzCollateral.sol:FuzzCollateral
echidna-test: Error running slither:
markspanbroek commented 1 year ago

Thanks for testing this earlier @emizzle! I have added a compile step to npm run fuzz to make sure that the contracts are compiled before invoking echidna. This makes the fuzzing work on a clean debian vm with only docker and nodejs installed. But I suspect it doesn't solve the problem that you're seeing on your system. Perhaps we can schedule a call to debug this further?

emizzle commented 1 year ago

Thanks for updating this @markspanbroek!

I have a new system since my last test unfortunately, and am running ARM64, so now I'm seeing the following error:

❯ npm run fuzz                    

> fuzz
> hardhat compile && fuzzing/fuzz.sh

Nothing to compile
Unable to find image 'ghcr.io/crytic/echidna/echidna:latest' locally
latest: Pulling from crytic/echidna/echidna
846c0b181fff: Pull complete 
a29861e18734: Pull complete 
929040e99d59: Pull complete 
33e42817120e: Pull complete 
8a58b4a70271: Pull complete 
Digest: sha256:82042765e513e3c5979d3edb3b46b8718b25bc2b48ea8971be76537a8cac5f8d
Status: Downloaded newer image for ghcr.io/crytic/echidna/echidna:latest
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Loaded total of 0 transactions from fuzzing/corpus/reproducers/
Loaded total of 0 transactions from fuzzing/corpus/coverage/
Analyzing contract: /src/contracts/FuzzCollateral.sol:FuzzCollateral
echidna-test: Error running slither:
markspanbroek commented 1 year ago

Rebased on latest commits in master.

emizzle commented 1 year ago

Thanks for updating this @markspanbroek!

I have a new system since my last test unfortunately, and am running ARM64, so now I'm seeing the following error:

❯ npm run fuzz                    

> fuzz
> hardhat compile && fuzzing/fuzz.sh

Nothing to compile
Unable to find image 'ghcr.io/crytic/echidna/echidna:latest' locally
latest: Pulling from crytic/echidna/echidna
846c0b181fff: Pull complete 
a29861e18734: Pull complete 
929040e99d59: Pull complete 
33e42817120e: Pull complete 
8a58b4a70271: Pull complete 
Digest: sha256:82042765e513e3c5979d3edb3b46b8718b25bc2b48ea8971be76537a8cac5f8d
Status: Downloaded newer image for ghcr.io/crytic/echidna/echidna:latest
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Loaded total of 0 transactions from fuzzing/corpus/reproducers/
Loaded total of 0 transactions from fuzzing/corpus/coverage/
Analyzing contract: /src/contracts/FuzzCollateral.sol:FuzzCollateral
echidna-test: Error running slither:

Looks like echidna doesn't support arm docker images, though they could be built with emulation. We could contribute to echidna a cross-platform ci solution that builds both amd64 and arm64 images.

markspanbroek commented 1 year ago

Looks like echidna doesn't support arm docker images

I added a check to the fuzz.sh script, so that it gives a more useful error message when running on arm systems.