search

zopefoundation / zope.sqlalchemy

Integration of SQLAlchemy with transaction management
Other
32 stars 34 forks source link

Coverage environment in Tox does not measure coverage accurately #80

Open Daverball opened 1 year ago

Daverball commented 1 year ago

The environment for coverage currently only is run on a specific Python/SQLAlchemy version and without the two Postgres specific test-runs on the CI, as such it does not accurately represent the coverage.

I have found that the best way to do coverage reports with tox and multiple environments is to set the environment variable that changes the name of the .coverage file, so each environment calculates the coverage and creates a unique file. Then at the end you run your report environment to stitch them together.

On the CI you would either have to create Artifacts and use them in a final dependent runner that stitches them together, or possibly coveralls supports stitching them together for you, if you upload multiple coverage reports from the same commit. (I know Codecov supports uploading multiple reports and stitches them together for you, but I have never used coveralls)

Daverball commented 1 year ago

Note that as part of #79 we ~possibly~ temporarily dropped the required coverage percentage ~or omitted the test code from test coverage~, so this should once again be raised/~re-enabled~ once this is fixed.

Daverball commented 1 year ago

Here is an example tox.ini from pytest-codecov: https://github.com/seantis/pytest-codecov/blob/master/tox.ini

Basically you are always measuring coverage for every test environment in a coverage file specific to that environment and all the report environment does is take the existing coverage files and combining them. pytest-cov makes this a little less cumbersome, but you could certainly accomplish the same results without it.

In the CI case with the additional testruns for postgres you would have to create two additional coverage files per environment. (Or four if you look at all permutations, but only ever two of them will be active in the same environment)

icemac commented 1 year ago

Coveralls also supports multiple uploads and combines them. To get a real coverage number only each action run needs to report to coveralls. This way it is not possible to get local coverage to a nice value but that is okay as locally we mostly do not run against an actual PostgreSQL installation but only the SQLite tests.