Code coverage in CI

[nevi-me]

It would be useful for us to have code coverage in CI, especially for the crypto backends. Depending on what service we use, this might require some integrating the repo to an external service, e.g. coveralls.

I can work on the Github Actions part and figure out how coverage will work for mutually exclusive targets, then I can leave some instructions for how a repo admin can enable CI on a service.

[simlay]

So, after a little bit of trying out cargo-llvm-cov, which I really like, I think this may require a bit of refactoring of CI should you want to use cargo-llvm-cov.

According to the README, one can merge different coverages based on features. The issue with this is that the different crypto backends are three different workflows (three links, for the three workflows, I'm not being passive aggressive or anything).

On a recent code sample I used codecov.io/, I was pleased how nicely it consumed the reports from cargo-llvm-cov with minimal CI addition. One can add the ability for it to comment on the PR but that could get annoying. I don't think using cargo-llvm-cov would force one into just using codecov.io though. The tooling has an --html option. It depends on how one wants the badge to be displayed.

Anyway, if it's reasonable, I'd suggest refactoring the three crypto backend CI's into one CI that:

• Uses a github job matrix for each backend
• Creates/uploads a github action artifact for each crypto backend of the coverage report steps
• A dependent job which downloads the report artifacts
• And finally Merge said coverage reports.
• Upload to codecov.io? Maybe put the content of cargo llvm-cov --html someplace?

[simlay]

I randomly found myself looking at this issue last week and decided to try out some new tools for my CI for work without relying on external websites like codecov.io (I think it was recently acquired). Some actions I found useful are:

• zgosalvez/github-actions-report-lcov will consume lcov files and comments on a PR with the results.
• livewing/lcov-job-summary will consume lcov files and create a Github Actions Job Summary.

This is an example of what I've got with merging multiple lcov's (from swift and rust tests) and turning them into a Job Summary