Code coverage (CI)

[ivirshup]

This applies to scanpy as well, but it's easier to try these things here since the test-suite runs faster.

There should be some kind of check on PRs for coverage. It should be really obvious if new code is tested, or if coverage goes down. There have a been a few cases in scanpy (mostly with external) where code just wasn't being tested on CI due to configuration issues. I'd like to avoid this.

Codecov

Codecov seems like a popular solution. I think we had it enabled at one point, but don't remember why we stopped. I remember finding it annoying, and that it gave a lot of notifications. I assume there are ways to make it less annoying, since I haven't been bothered by it recently when I contribute to repositories with it on.

It would be pretty easy to turn on, just add this to test coverage jobs.

It seems very configurable, but without a huge amount of documentation.

Options for integrations include:

• a bot comment on a PR with coverage information that gets updated
• Using github checks

Features I'd like

• I'd like to be able to see which test covered which lines. I don't remember seeing this happen with codecov, but maybe they allow this now. We can generate this output with contexts.
• A diff of coverage for a PR
• A badge for the repo
• Not necessarily failing a PR if some coverage isn't met, but definitely making it visible

[flying-sheep]

Sure, let’s tackle all this directly after #478!

[ivirshup]

As noted by @flying-sheep in https://github.com/theislab/scanpy/issues/1576#issue-783385380, there is some support for this directly in azure via an azurepipelines-coverage.yml config file. Unfortunately it looks like it's not available for python projects yet:

While you can collect and publish code coverage results for many different languages using Azure Pipelines, the code coverage for pull requests feature discussed in this document is currently available only for .NET and .NET core projects using the Visual Studio code coverage results format (file extension .coverage). Support for other languages and coverage formats will be added in future milestones.