Track code coverage metrics

[jswrenn]

As part of upholding our design ethos (#405), we would like to be able to promise that we maintain exceptionally high code coverage. To do this, we need to track code coverage in CI.

Mentorship Instructions

We're unfamiliar with the best-practices of monitoring code coverage in Rust. We are hoping that the contributor to this issue can help us navigate the complexities of applying code coverage metrics to this repo, which contains:

• multiple packages (namely, zerocopy and zerocopy-derive)
• unit tests
• integration tests
• ui tests

In CI, these tests are run in multi ways (e.g., via miri, kani, and binary) on multiple platforms. Ideally, our coverage metrics captures as many of these sources of coverage as possible.

Unresolved Questions

• How should we report these metrics on PRs?

[tommy-gilligan]

I'm keen to take this. I've used tarpaulin a bit before.

How should we report these metrics on PRs?

My go-to here would be Github Actions artifacts.

[joshlf]

That'd be awesome; it's all yours! I've assigned it to you.

Github Actions artifacts sounds reasonable for reporting the metrics.

[tommy-gilligan]

Maybe I spoke too soon recommending Tarpaulin. Nevertheless, here is an example build using Tarpaulin https://github.com/tommy-gilligan/zerocopy/actions/runs/7986802695 .

I think I'll also try cargo-llvm-cov

Regardless of which tool is used, would it be useful to aggregate across different axes of ci.yml matrix? What kind of aggregations would be useful?

PS. I'm finding navigating workflow jobs a bit tricky because the (necessarily) long names get truncated. I'm curious if you have a handy way of accessing the same information in a more accessible way?

[joshlf]

Maybe I spoke too soon recommending Tarpaulin. Nevertheless, here is an example build using Tarpaulin https://github.com/tommy-gilligan/zerocopy/actions/runs/7986802695 .

Where is the coverage information reported there? I don't see it.

@jswrenn is the maintainer of itertools; here's their configuration: https://github.com/rust-itertools/itertools/blob/master/.github/workflows/coverage.yml

I don't personally have experience with this, so I can't promise that some or any of it will be applicable here, but it might be a useful place to start 🤷‍♂️

Regardless of which tool is used, would it be useful to aggregate across different axes of ci.yml matrix? What kind of aggregations would be useful?

Here's a rough outline of what we'd want in order of priority (highest priority to lowest). Let's start with the first and then we can try for the lower ones if it turns out to be feasible.

• Report coverage only from a particular matrix combination (in particular, Linux x86_64 with --all-features on the nightly toolchain)
• Report coverage from all matrix combinations which have the Linux x86_64 target
• Report coverage from all matrix combinations

For the first of these, there's no need to solve the "do we report separately or aggregate" question since there's just one combination being reported.

PS. I'm finding navigating workflow jobs a bit tricky because the (necessarily) long names get truncated. I'm curious if you have a handy way of accessing the same information in a more accessible way?

Unfortunately none that I'm aware of, but I also haven't tried hard to find one.

[joshlf]

I added basic coverage in #1274. There are still a lot of improvements that could be made, especially support for zerocopy-derive.