Coverage of non-forge tests for solidity contracts

[Inphi]

Some contracts in our repository (e.g., MIPS64.sol) are tested outside of forge in a go-ethereum (geth) execution environment.

For Cannon contracts specifically, an EVM is instantiated with pre-deployed contracts using forge-generated artifacts. Cannon uses differential testing to verify that the Solidity and offchain VMs produce identical results. These differential tests are implemented as standard Go tests using the EVM setup described above. This is done for ease of writing tests as Go is a much faster and more expressive than forge for testing VMs. The downside of using Go is that we don't have a way to easily figure out how much of the Solidity VMs these Go tests cover.

To solve this, we need a Go tool/package that can:

1. Take forge/solc artifacts and an EVM execution trace as input
2. Generate lcov-formatted coverage data as output

Ideally, this package is implemented as an EVM tracing hook so that it can easily be slotted into the cannon EVM driver.

This package can then be later used to generate coverage data from Cannon's Go tests. However, the follow up is out of scope of this ticket.

[smartcontracts]

Wondering if one option here would be to do the differential testing in Solidity instead? We'd get the coverage for free that way.

[Inphi]

Wondering if one option here would be to do the differential testing in Solidity instead? We'd get the coverage for free that way.

We attempted this for the OG MIPS.sol tests but it is too slow to run differential tests in solidity. The ffi overhead is no joke. Particularly once you have fuzz tests. I actually looked into speeding up ffi in forge some time back using a client-server model, but didn't get anywhere.

[Inphi]

A couple considerations. The source maps found in forge artifacts aren't perfect. Code inlining, common function elimination, and a couple other compiler tricks make it non-trivial to corroborate program counters in the EVM with the source code. It would be useful to take a look at this source map tracer to figure out how it handles some broken cases in source maps. But even if the coverage data isn't perfect, it's still better than zero coverage. And if there's a way we can rewrite contracts or use certain solc compiler flags to make it easy to generate coverage, it'll be worth doing.

[JosepBove]

I'll circle around this issue and come back with some ideas.

[smartcontracts]

We should do this in coordination with @AmadiMichael who is making sure that we are able to compile the contracts without the optimizer enabled, which will make coverage easier.