CI: Test coverage reporting

[cpelley]

Avoid the pitfalls of another library that dagrunner sets to replace. Mainly, that we don't end up with poor test coverage down the line. To that end, it would be ideal if we could utilise a single approach that could be used irrespective to whether our repository is public or private.

To that end, we avoid the use of external services such as codecov.

Here, we run test coverage on both the PR and base branch (main, if it hasn't run against the SHA of the base before). We then compare their total percentage test coverage and if the PR results in a total percentage reduction in coverage of greater than 1% (roundup tolerances), comment on the PR to draw attention to this reduction.

• Cache utilisation for pip.
• Fix for existing pre-commit cache utilisation.
• Test coverage comparison against the base branch (main).
  • Checkout and run testing against the base branch ONLY if the base commit SHA has changed since its last run, otherwise simply recall the report cache for comparison against the one we generate for this PR.
  • When percentage coverage has reduced compared to the base branch:
  • Leave comment on the PR showing what this reduction is and with guidance.
  • Upload the summary report and html full report via artefact.
• Updated left over action syntax deprecation (set-output).

Issues

• https://github.com/MetOffice/improver_suite/issues/2214

Testing

Demonstrated by the inclusion of a dummy python dummy module to expose the change in coverage (see comment: https://github.com/MetOffice/dagrunner/pull/38#issuecomment-2203166049)

To look at the full report, simply download the uploaded artefact. For example, for the job/26943990384 job run

We can see the text summary has been recorded alongside a html report:

We can open the html report with our browser by opening the index.html:

We then have total control to analyse our coverage report, take for example when we click on the execute_graph.py module:

Note

Obviously we can add as much coverage drop context as we wish to the PR comment, but I have opted for keeping it minimal and to the point (at least in the first instance).

[github-actions[bot]]

The test coverage has decreased from '81%' to '71%' (commit SHA: 4bf3841f9235fa7f2b649687d72423af6d1a11cf). Please review test coverage. Report uploaded as artifact.

[cpelley]

Another one of likely interest to you @s-boardman for StaGE. This enabled test coverage reporting without the usage of external services. This means we can use whether our repository is public or private. Obviously one can enhance the comment indicating coverage (I just did basic working concept for this PR).