Missing code-coverage within `pytest_sessionfinish`

[dshah244]

Summary

Code present within project.specific.conftest::pytest_sessionfinish() is shown as "not covered", although it is executed.

Comparing pytest-cov with coverage,

Code within pytest_sessionfinish is presented as covered when python-coverage is used with the following command

coverage run --include conftest.py -m pytest test_dummy.py && coverage html -d cov_report

Link to test_dummy.py.

Expected vs actual result

Code executed within pytest_sessionfinish should be presented as covered.

Reproducer

README files consists of the instructions to setup the virtual-environment and pytest framework.

Other files of interest within the referred repository:

• pyproject.toml: consists of pytest-cov related switches

Versions

Requirements file can be accessed here: https://github.com/dshah244/dummy_pytest/blob/master/req_dev.txt

Output of pip freeze:

-e .
iniconfig==1.1.1
packaging==21.3
pluggy==1.0.0
py==1.11.0
pyparsing==3.0.6
pytest==6.2.5
pytest-cov==3.0.0
pytest-html==3.1.1
pytest-metadata==1.11.0
toml==0.10.2
tomli==1.2.2

python --version= pytest 6.2.5

pytest --version= Python 3.10.0

Config

https://github.com/dshah244/dummy_pytest/blob/master/pyproject.toml

Code

https://github.com/dshah244/dummy_pytest

[gitpushdashf]

What version of coverage are you using? I noticed some strange behavior going from 6.1.2 to 6.2, where lines that are executed weren't covered.

[dshah244]

Hi,

Nice to hear from you :). Upon using pip list it does turn out that coverage==6.1.2 is being used, which might explain your comment

But on the other hand it doesn't explain why using pytest-cov results in a different coverage as compared to using vanilla coverage for code coverage.

My assumption here is that pytest-cov simply wraps the available vanilla coverage functionality.

Full pip list output is as follows

Package         Version Editable project location
--------------- ------- --------------------------------------
attrs           21.2.0
coverage        6.1.2
dummy-framework 0.1.0   .
iniconfig       1.1.1
packaging       21.3
pip             21.3.1
pluggy          1.0.0
py              1.11.0
pyparsing       3.0.6
pytest          6.2.5
pytest-cov      3.0.0
pytest-html     3.1.1
pytest-metadata 1.11.0
setuptools      59.2.0
toml            0.10.2
tomli           1.2.2

[dshah244]

The coverage results on the right are generated using the vanilla coverage commands and the results on the left are generated by using pytest-cov.

I have also uploaded the coverage report to my dummy github project

• cov_report_pytest
• cov_report_coverage

and added the instructions to generate such coverage reports within README

[nedbat]

@gitpushdashf did you report that issue?

What version of coverage are you using? I noticed some strange behavior going from 6.1.2 to 6.2, where lines that are executed weren't covered.

[nedbat]

@dshah244 you might try adding tryfirst=True, so that your code runs before pytest-cov does its cleanup. Or, why not use the vanilla coverage commands?

[dshah244]

Hi @nedbat ,

Thanks for the suggestion (y) I tried tryfirst=True and unfortunately it also doesn't seem to work.

"""Or, why not use the vanilla coverage commands?""" - What I like about pytest-cov is that it dynamically assigns coverage-context based on the tests and test-phases (setup, call, teardown), even to pytest-fixtures being called.

This is not possible using vanilla coverage, specially the bit where pytest-fixtures are assigned with a context. Vanilla coverage, simply, does not assign any context to the pytest-fixtures being called by test-cases + the context are limited to only the test-cases and not test-phases.

[gitpushdashf]

@gitpushdashf did you report that issue?

I did not. Unfortunately, it's in a proprietary codebase that's moderately large and complex. I was hoping someone else might run into it on something smaller and easier to narrow it down with.

Edit: It looks like concurrency = multiprocessing was the issue. I removed that and it runs fine with 6.2. Not sure about 6.1.2, if it's needed or what not.

[dshah244]

Hi everyone,

Just a comment to keep the thread alive and in spotlight. This fix would be a big convenience plus it would improve consistency in how pytest-cov behaves as compared to coverage itself.

Pros:

• pytest-cov provides test(-setup, -call, -teardown) as coverage-contexts out-of-the-box and these could be seen in the coverage-report generated by it. Such a behaviour is not possible by using vanilla coverage (out-of-the-box).

For now, I have simply decided to use vanilla coverage (I'll try to dig a bit deeper myself when time permits).

And lastly, wishing you the best for the new year : )

[ionelmc]

You might want to copy some of the tricks here: https://pytest-cov.readthedocs.io/en/latest/plugins.html

[dshah244]

You might want to copy some of the tricks here: https://pytest-cov.readthedocs.io/en/latest/plugins.html

Thanks for the tip @ionelmc 👍

[stefanofusai]

Hey @dshah244, I'm experiencing the same issue. Did you manage to solve this?