Improve fail_under by enabling more refined control on threshold

nedbat / coveragepy

The code coverage tool for Python

https://coverage.readthedocs.io

Apache License 2.0

2.99k stars 429 forks source link

Improve fail_under by enabling more refined control on threshold #666

Open nedbat opened 6 years ago

nedbat commented 6 years ago

Originally reported by Anonymous

Hello,

I didn't find this option anywhere in the documentation so i hope i haven't missed anything. The idea is that right now we can only set a global minimum threshold percentage on coverage. While this is really useful i suggest coverage could provide a more refined control on this requirement such as per module minimum percentage.

Taking for example jest (https://facebook.github.io/jest/docs/en/configuration.html#coveragethreshold-object) it would be really great if i could set in the report section my requirements like maybe a list of regexes with minimum requirements.

api/* -> 40%
db/* -> 80%

Thanks in advance

Bitbucket: https://bitbucket.org/ned/coveragepy/issue/666

nedbat commented 6 years ago

Original comment by Orestis Ioannou (Bitbucket: oorestisime, GitHub: oorestisime)

ok right, if you are nonetheless willing to let this go in coverage i could have a look on how to implement this and propose a PR. otherwise i guess i should close the issue :)

nedbat commented 6 years ago

I don;t have a specific tool in mind. It could even be a new tool that you run after coverage, to make a detailed assessment of the results. It doesn't have to be added to anything existing.

nedbat commented 6 years ago

Original comment by Orestis Ioannou (Bitbucket: oorestisime, GitHub: oorestisime)

Maybe this could be added to pytest-cov although i am not sure whether they would want this or if it is better to support this kind of feature directly in coverage so that all the runners could benefit from it.

nedbat commented 6 years ago

Original comment by Orestis Ioannou (Bitbucket: oorestisime, GitHub: oorestisime)

I guess so yes, do you have a tool in mind?

nedbat commented 6 years ago

Interesting idea, but it seems very specialized for something like coverage. A separate tool could look at the coverage report, and make this kind of assessment. Would that work for you?

nedbat commented 6 years ago

A JSON report (#720) would make it possible to implement this feature, especially if it included sub-totals for each directory of code.

nedbat commented 2 years ago

See https://nedbatchelder.com/blog/202111/coverage_goals.html for some progress on a tool to do this.