CosmWasm test coverage tool is needed

[SteMak]

Background

Covering code with tests is an important part of smart contract development, but, unfortunately, there is no documented way to run a coverage check, so some critical branches coverage may be missed. When smart contract logic become complex, it is not possible to manually check if all cases are covered with tests. It happens 1000 lines of tests per 200 lines of code, and it is easy to miss some critical points in such a huge amount of code.

Popular tool tarpaulin don't work intuitively

For now, I'm receiving

Error: "Failed to get test coverage! Error: Error while parsing"

when running

cargo tarpaulin --lib

However, the project is built successfully.

Solution

Provide documentation of how to check tests code coverage. Possibly tarpaulin tool is working, but should be configured, then provide configuration of it.

I hope, there is a way to check tests CosmWasm code coverage, please share it to community.

We want to write smart contracts with 100% coverage!

[ethanfrey]

It seems pretty straight forward. I just used tarpaulin out of the box.

This is how we run it in CI as well: https://github.com/CosmWasm/cw-plus/blob/main/.circleci/config.yml#L703-L716

Hope this helps

[SteMak]

Thank you for your notice!

Yeap, it works okay when running

cargo tarpaulin --skip-clean --frozen --out Html --output-dir coverage --lib -- --color=always

However, I'm not sure about integration tests coverage. Is it generally possible to calculate the coverage for integration tests? When I'm trying to run something like

cargo tarpaulin --skip-clean --frozen --out Html --output-dir coverage --test integration -- --color=always

I got 100% coverage of the tests/integration.rs file and zero coverage for the src folder.

As it is never documented of how to run tarpaulin for CosmWasm, leave the issue open.

[ethanfrey]

Ah, integration tests can't work, as they are loading a wasm image of the file that is being tested and there is no way to instrument which lines of the original rust contract is called.

It works great for unit tests and multi-tests (basically anything that doesn't involve wasm)

[uint]

Just to add, I think if you look at it from a testing philosophy POV, I'm not sure measuring coverage makes sense for higher layers of testing than UTs anyway.

With integration/system/whatever testing, your goal is no longer to ensure all code logic works as expected. It instead becomes stuff like "is this interface behaving as expected", "is this component/system's behavior what we expect", "does this set of components play well together", and "are all our business requirements fulfilled", with your contract treated as a black box.

[SteMak]

I'm closing the issue as cargo tarpaulin normally calculates coverage for unit tests I don't remember, why I had issues with running it out-of-box when I opened the issue

I hope, someday the "test coverage" topic will be mentioned in the documentation