[Bug] Aptos CLI does not pipe through Move CLI coverage testing

[alnoki]

🐛 Bug

The Move CLI, on which Aptos CLI is based, offers coverage testing via move package test --coverage, allowing developers to verify the output of all conditional logic branches in a module. Hence pure-Move implementations can be tested to software design assurance standards like DO-178B Level A, a design specification intended to prevent catastrophic system failure, by assuring 100% code coverage. For example, Econia::CritBit is tested to 100% coverage, which means that every output of every conditional statement therein has been verified by a test.

The Aptos CLI, however, offer no coverage testing support, and since the Move CLI does not offer AptosFramework native function support, Aptos developers are thus prohibited from running coverage tests on non-pure-Move implementations (e.g. modules which use AptosFramework modules, like Econia::Market). This is a serious security vulnerability: if developers cannot test to 100% coverage, then they cannot verify that all of their code is actually behaving as intended. Hence without coverage testing there is an increased likelihood of catastrophic system failure (e.g. loss of funds), and as such it is advised that the Aptos CLI be updated to pipe through Move CLI coverage testing. This will allow developers to run coverage testing on code using AptosFramework native functions, and to thus verify that all of their code actually behaves as expected.

Split off from #1256

[gregnazario]

We can add the coverage from the Move CLI directly into the Aptos CLI. In general, I've been looking at improving the user experience. I was asking around to others if it was possible to tell us what were the actual lines that were covered similar to the tools I was used to in other languages like Java.

[alnoki]

@gregnazario Right now the Move CLI does offer this functionality: after running move package test coverage, then running move package coverage summary will print out a percentage for each file, and running move package coverage source --module <module_name> will print the coverage map with covered lines in green and uncovered lines in red. But I've noticed that sometimes the coverage map file is too large to render in a terminal (at least the VS Code integrated terminal). Feel free to check out these commands using the CritBit module I linked to above (e.g. you can delete a test or two to get rid of some coverage)

I would definitely prefer an interactive coverage viewing tool, though, perhaps in the style of a VS Code extension?

[gregnazario]

Yeah, I was thinking this would probably provide the backing for such an extension

[leofisG]

Is supporting coverage testing in the pipeline? It will be a valuable feature precisely as @alnoki mentioned.

[wrwg]

Yes, we will come to this soon.

[github-actions[bot]]

This issue is stale because it has been open 45 days with no activity. Remove the stale label or comment - otherwise this will be closed in 15 days.

[leofisG]

@wrwg any update on this?

[wrwg]

Sorry for the delay. This should come now really soon ;-)

[github-actions[bot]]

This issue is stale because it has been open 45 days with no activity. Remove the stale label or comment - otherwise this will be closed in 15 days.