search

sc-forks / solidity-coverage

Code coverage for Solidity smart-contracts
MIT License
961 stars 262 forks source link

feature request: hybrid Foundry/Hardhat coverage #734

Open wschwab opened 1 year ago

wschwab commented 1 year ago

solidity-coverage is currently a framework agnostic project. While Foundry is working on its own coverage tool, I think that there is value in extending solidity-coverage's support to Foundry too. Once specific use case follows.

Certain things are easier to test from JS/TS as an EOA. Some are easier to test as a smart contract (on-chain). Hardhat provides the former, Foundry provides the latter (and easy access from within tests to fuzz and invariant tests). Some repos have decided on a hybrid environment, bringing in features from both. Seaport is a prominent example.

Getting coverage results that include the coverage of both the JS/TS and Solidity tests would be where solidity-coverage could shine. Once the tool would be able to monitor coverage from Foundry, it would seem to me that it wouldn't be too hard for it to monitor overall coverage in a hybrid repo.

If there's anything I can do to add context and/or information, please let me know.

Am I looking at this right?

cgewecke commented 1 year ago

@wschwab

Once the tool would be able to monitor coverage from Foundry

I don't think this is likely to happen (or desirable). solidity-coverage is tightly coupled to the EthereumJS VM and works best with JS toolchains.

Maybe hybrid repo users could just merge saved reports from different test runs using a tool like istanbul-combine-updated. You can select what kind of report to produce in solidity-coverage using the istanbulReporter option (IIRC, Foundry generates a standard LCOV artifact).

If you're using Codecov as a CI service, multiple reports uploaded at once will be auto-merged on their side.

Does this approach seem reasonable for your use case?

wschwab commented 1 year ago

Thanks for the detailed answer @cgewecke - we're looking into implementing a solution using your answer. If you don't mind, I'll keep this open in the meantime just in case we run into trouble.

yorhodes commented 1 year ago

https://github.com/hyperlane-xyz/hyperlane-monorepo/blob/main/solidity/coverage.sh we merge the two reports here

yorhodes commented 1 year ago

I spent ages fiddling with this but hands down codecov is the best solution by far https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2449 free for open source!