Estimate code coverage for the current server codebase

[shankari]

The current server codebase https://github.com/e-mission/e-mission-server has automated tests https://github.com/e-mission/e-mission-server/tree/master/emission/tests and continuous integration https://github.com/e-mission/e-mission-server/tree/master/.github/workflows

We would like to now estimate the code coverage of this test suite, with a view towards improving it in the future. A potential code coverage tool is https://github.com/nedbat/coveragepy, but we should look at other options as well.

Ideally, the selected tool would have a GitHub action that we could use to compute the code coverage. We could display the current code coverage as a button (see below)

and even the difference as part of a pull request - e.g. something like https://josh-ops.com/posts/github-code-coverage/

This will allow us to maintain the coverage over time.

[shankari]

@aGuttman, can you update this with the design decisions behind the choice of the code coverage tool? You could also document your findings as you explore the tool and choose config options so we know why the options were chosen.

[shankari]

@aGuttman Here's a tool that converts code coverage reports into human readable text https://reportgenerator.io/ It also seems to have a list of code coverage tools :)

[aGuttman]

• Coverage.py: https://github.com/nedbat/coveragepy
  • Easy to install, use
  • Seems to be the go-to choice for python coverage, good support, information online
  • Reasonable, readable command line reports
  • Detailed, readable html reports
  • Additional GitHub action support, ie: https://github.com/marketplace/actions/python-coverage-comment
  • Badges for repo readme
  • Coverage diff reports in pull requests
• Report Generator: https://reportgenerator.io
  • Rich visualizations
  • Support for GitHub actions: https://github.com/marketplace/actions/reportgenerator
  • Only generates reports, is not a coverage tool itself
  • Mostly built to support .NET coverage tools, also has support for some Java, C, C#, no python
• pytest-cov: https://pytest-cov.readthedocs.io/en/latest/readme.html
  • Plug-in that uses coverage.py
  • Auto-configures multiprocessing settings
  • Might be easier to use in the long run but I can't tell. I understand how to do what I currently want with coverage.py directly, I'm confused about how to do it in pytest-cov
• Code coverage in PyCharm
  • JetBrains product, not allowed to use?

It seems that basically everyone uses coverage.py/pytest-cov for python. Difficult to even find information about what else is out there.

[aGuttman]

Actions options:

• Codecov: https://about.codecov.io/blog/python-code-coverage-using-github-actions-and-codecov/

  • Seems really slick, nice visuals, but don't know how much use out of charts, etc provided
  • Sign-up required
  • Paid product, free for open source

• Coveralls: https://github.com/marketplace/actions/coveralls-python

  • Similar to Codecov, rich visuals, sign up required, "freemium"

• Community options

  • https://github.com/marketplace/actions/python-coverage-comment
  • Can be used to comment coverage changes on pull requests and display badge on readme
  • https://github.com/tj-actions/coverage-badge-py
  • very simple implementation for displaying badge
  • https://github.com/marketplace/actions/pytest-coverage-commentator
  • Detailed comments on pull requests

• Hack it up ourselves

  • https://itnext.io/github-actions-code-coverage-without-third-parties-f1299747064d
  • https://dedreira.medium.com/create-a-github-actions-ci-pipeline-for-a-python-package-4c4c02dc5f2e
  • https://hynek.me/articles/ditch-codecov-python/

I've barely used GitHub actions. I can get the simple stuff to work with my current knowledge, might need time to understand more involved actions but it does't look too bad. The community options look good to me, I don't feel like we need the fancy charts, and are still easy to use.

[shankari]

@aGuttman I agree with both your recommendations:

• coverage.py/pytest-cov for the actual code coverage; it looks like we don't have a lot of choice anyway
• community option for GitHub actions. I agree that they are good enough for what we need right now. We can always upgrade to the "freemium" version later. Note that we probably need to get approval before using a third-party commercial tool anyway.

Can you implement at least the first step and get the current code coverage before our meeting with the cloud folks on Monday?

[aGuttman]

Yes, definitely. Coverage is easy to use and I've been playing around with different actions in my fork so it all basically works already, I just need to settle on an implementation and hammer out any details along the way.