Originally posted by **mattkwiecien** January 9, 2024
Today we discussed edge cases where the total test coverage metric may decrease when either no new code is added or when code quality is increased. In these instances the CI would reject the PR even if the PR contained improvements. For these reasons we want to investigate better test coverage metrics for firecrown.
We discussed two features that would be nice to have from code coverage:
1. All _new_ lines of code added in the PR should be covered by unit tests
2. Any _changed_ lines of code in the PR should be covered by unit tests
It doesn't appear that `coveralls` supports either of these checks and only supports a total code coverage metric/check for CI. There are other tools which do support this. I've found the two most popular appear to be `diff-cover` and `Codecov`, though `diff-cover` seems more finnicky and has less documentation.
`Codecov` appears much more widely used with extensive documentation, and easier to set up, but would involve moving away completely from `coveralls`. With `Codecov`, we could implement both of the above features using `Codecov/patch` and `Codecov/project` (see github check details here https://docs.codecov.com/docs/github-checks). Is this something we would consider?
Add codecov
Create a working implementation of differential code coverage with codecov
As a final step, switch from coveralls to codecov.
Discussed in https://github.com/LSSTDESC/firecrown/discussions/354