Increase threshold for failed code coverage checks

alyssadai commented 1 month ago

Let's use https://about.codecov.io/ consistently across repos and also increase threshold for failed checks.

At the moment, our code coverage checks (e.g., via coveralls) fail when coverage decreases by any amount. For example, see https://github.com/neurobagel/api/commit/a93d476f4ffbfd6ab7413de5c1fba7f92882c0a4 which failed the coverage check due a 0.7% decrease.

To avoid possibly superfluous tests just for the sake of ensuring 0% coverage decrease, I propose we bump up the threshold so that coveralls / codecov only complains when coverage has decreased 1% or more.

Not doing now:

[ ] https://github.com/neurobagel/query-tool/issues/62

Because that requires a larger setup of testing coverage in the first place.

surchs commented 1 month ago

Could we make it relative to the PR rather than the codebase? I.e. we allow up to X% (maybe 20%?) of new or changed lines in a PR to be uncovered? % of the whole codebase is a bit harder to judge for me

surchs commented 1 week ago

I configured the global settings here: https://app.codecov.io/account/gh/neurobagel/yaml

coverage:
  status:
    patch:
      default:   # default is the status check's name, not default settings
        target: auto 
        threshold: 5

My understanding is that this will apply a 5% threshold on patches: https://docs.codecov.com/docs/commit-status

alyssadai commented 1 week ago

Hey @surchs, just thought of something: would you mind in your PRs also updating the README badges for coverage in the repos that have it? Currently they use coveralls ones I believe.

neurobagel / planning

Increase threshold for failed code coverage checks #173