search

foundry-rs / foundry

Foundry is a blazing fast, portable and modular toolkit for Ethereum application development written in Rust.
https://getfoundry.sh
Apache License 2.0
8.22k stars 1.71k forks source link

bug: expecting build reproducibility for `forge test`, `forge build` and `forge coverage` #8889

Open gnapoli23 opened 1 month ago

gnapoli23 commented 1 month ago

Component

Forge

Have you ensured that all of these are up to date?

What version of Foundry are you on?

forge 0.2.0 (27d008f 2024-09-07T11:27:46.212820056Z)

What command(s) is the bug in?

forge test - forge build - forge coverage

Operating System

Linux

Describe the bug

I have a Foundry project which builds, tests and creates test coverage in a docker container based on the latest version of ghcr.io/foundry-rs/foundry image.

The project is hosted in an internal repository, and I expect that when I run forge build then running forge test would not recompile the contracts, but in many cases it happens that even if I don't make any change to the contracts source code, other colleagues running forge test on their machines have a recompilation even if they are "pre-compiled" and versioned in the repo.

This does not happen always, is there a reason behind that?

More over, we noticed that when we run forge coverage the recompilation always happens, indeed we have diffs in the compiled contracts. The diffs seems to be in the formatting of the file, btw.

Is there a way to have build reproducibility for these 3 commands whatever is the machine we are running them on? Is the compilation somehow based on the timestamp of the files? Thanks for your support.

zerosnacks commented 1 month ago

Regarding forge coverage, it requires optimizations to be disabled (which is enabled by default, see https://github.com/foundry-rs/foundry/issues/2486) so recompilation is expected here.

Does running forge test require a full recompilation or a partial recomplilation?

gnapoli23 commented 1 month ago

Hi @zerosnacks , thanks for the info about forge coverage.

Regarding forge test usually it's a partial recompilation, not a full one.

SocksNFlops commented 3 weeks ago

This is giving weird behaviors when utilizing the creationCode of contracts in lib dependencies. In other words, it's impossible to use precomputed hashes. For instance, if you run this in forge test and forge compile, you will get different results:

import {CONTRACT} from ...
...
bytes32 initHashCode = keccak256(abi.encodePacked(type(CONTRACT).creationCode));
console.log("initHashCode");
console.logBytes(initHashCode);

If you pre-compute them and hardcode them in the contracts, forge test works fine.

However, if you run forge coverage, it will recompile them with different creationCodes than the ones used in forge test, and these won't match the precomputed ones anymore. This breaks all of the unit tests where you're attempting to use the hash to predict a create2 output.

What's even worse, is after you run forge coverage, it caches the build of the lib-dependencies, and then proceeds to break every run of forge test unless you run forge clean.

I believe this is because it's recompiling the lib-dependencies without optimizations, whereas in unit tests and deployments, they have optimizations on. If forge coverage is going to recompile everything in src/, is there a way we can at least allow our lib dependencies to be compiled with the standard optimizations? Left as is, either forge test or forge coverage will run successfully, but not both. This isn't ideal.