search

pytest-dev / pytest-cov

Coverage plugin for pytest.
MIT License
1.76k stars 211 forks source link

Test coverage not narrowed to package source #402

Open lexicalunit opened 4 years ago

lexicalunit commented 4 years ago

Summary

When I run coverage directly I get a sensible report showing a coverage of my module at 83%:

$ cat .coveragerc
[run]
source = turbot
omit =
    scripts/*
    setup.py

$ coverage run -m pytest
================================================================================================================================================================================ test session starts ================================================================================================================================================================================
platform darwin -- Python 3.7.6, pytest-5.4.1, py-1.8.1, pluggy-0.13.1
rootdir: /Users/me/source/turbot
plugins: asyncio-0.10.0, mock-3.0.0, freezegun-0.4.1, cov-2.8.1
collected 86 items

tests/test_turbot.py ......................................................................................

$ coverage report -m
Name                 Stmts   Miss  Cover   Missing
--------------------------------------------------
turbot/__init__.py     661     83    87%   27-28, 84, 133, 147, 217-281, 386, 820, 879, 1080-1087, 1092-1096, 1153-1158, 1169
turbot/migrate.py       35     35     0%   3-50
--------------------------------------------------
TOTAL                  696    118    83%

However when I use pytest-cov with tox I get pure madness:

$ tox
GLOB sdist-make: /Users/me/source/turbot/setup.py
clean installed: coverage==5.1,turbot==2.0.0
clean run-test-pre: PYTHONHASHSEED='818505583'
clean run-test: commands[0] | coverage erase
py37 inst-nodeps: /Users/me/source/turbot/.tox/.tmp/package/1/turbot-2.0.0.zip
py37 installed: aiohttp==3.6.2,appdirs==1.4.3,async-timeout==3.0.1,attrs==19.3.0,beautifulsoup4==4.9.0,black==19.10b0,callee==0.3.1,certifi==2020.4.5.1,chardet==3.0.4,click==7.1.1,coverage==5.1,cycler==0.10.0,discord.py==1.3.3,distlib==0.3.0,entrypoints==0.3,filelock==3.0.12,flake8==3.7.9,freezegun==0.3.15,idna==2.9,importlib-metadata==1.6.0,isort==4.3.21,kiwisolver==1.2.0,lxml==4.5.0,matplotlib==3.2.1,mccabe==0.6.1,more-itertools==8.2.0,multidict==4.7.5,numpy==1.18.2,packaging==20.3,pandas==1.0.3,pathspec==0.8.0,pluggy==0.13.1,py==1.8.1,pycodestyle==2.5.0,pyflakes==2.1.1,pyparsing==2.4.7,pytest==5.4.1,pytest-asyncio==0.10.0,pytest-cov==2.8.1,pytest-freezegun==0.4.1,pytest-mock==3.0.0,python-dateutil==2.8.1,pytz==2019.3,PyYAML==5.3.1,regex==2020.4.4,requests==2.23.0,six==1.14.0,soupsieve==2.0,toml==0.10.0,tox==3.14.6,turbot==2.0.0,typed-ast==1.4.1,urllib3==1.25.9,virtualenv==20.0.18,wcwidth==0.1.9,websockets==8.1,yarl==1.4.2,zipp==3.1.0
py37 run-test-pre: PYTHONHASHSEED='818505583'
py37 run-test: commands[0] | pytest --cov=. --cov-report=term-missing --cov-report html --no-cov-on-fail
================================================================================================================================================================================ test session starts ================================================================================================================================================================================
platform darwin -- Python 3.7.6, pytest-5.4.1, py-1.8.1, pluggy-0.13.1
cachedir: .tox/py37/.pytest_cache
rootdir: /Users/me/source/turbot
plugins: asyncio-0.10.0, mock-3.0.0, freezegun-0.4.1, cov-2.8.1
collected 86 items

tests/test_turbot.py ......................................................................................                                                                                                                                                                                                                                                                   [100%]

================================================================================================================================================================================= warnings summary ==================================================================================================================================================================================
.tox/py37/lib/python3.7/site-packages/pytest_asyncio/plugin.py:39: 82 tests with warnings
  /Users/me/source/turbot/.tox/py37/lib/python3.7/site-packages/pytest_asyncio/plugin.py:39: PytestDeprecationWarning: direct construction of Function has been deprecated, please use Function.from_parent
    item = pytest.Function(name, parent=collector)

.tox/py37/lib/python3.7/site-packages/pytest_asyncio/plugin.py:45: 82 tests with warnings
  /Users/me/source/turbot/.tox/py37/lib/python3.7/site-packages/pytest_asyncio/plugin.py:45: PytestDeprecationWarning: direct construction of Function has been deprecated, please use Function.from_parent
    item = pytest.Function(name, parent=collector)  # To reload keywords.

-- Docs: https://docs.pytest.org/en/latest/warnings.html

---------- coverage: platform darwin, python 3.7.6-final-0 -----------
Name                                                                                    Stmts   Miss  Cover   Missing
---------------------------------------------------------------------------------------------------------------------
.tox/py37/bin/black                                                                         6      0   100%
.tox/py37/bin/flake8                                                                        6      0   100%
.tox/py37/bin/isort                                                                         6      0   100%
.tox/py37/lib/python3.7/site-packages/_black_version.py                                     1      0   100%
.tox/py37/lib/python3.7/site-packages/_pytest/_argcomplete.py                              37     36     3%   56-106, 109
.tox/py37/lib/python3.7/site-packages/_pytest/_code/code.py                               654    633     3%   1-51, 61-71, 76, 81, 83-1145, 1154-1156, 1165-1188
.tox/py37/lib/python3.7/site-packages/_pytest/_code/source.py                             244    221     9%   1-36, 40-282, 295, 302-303, 310-311, 322, 325-326, 332-416
.tox/py37/lib/python3.7/site-packages/_pytest/assertion/__init__.py                        79     69    13%   4-103, 113, 139-150, 157-162, 169-179
.tox/py37/lib/python3.7/site-packages/_pytest/assertion/rewrite.py                        595    517    13%   2-67, 72-74, 76, 97, 111, 114, 133-134, 142-149, 154, 183, 188, 193, 196-197, 201-204, 216, 222-226, 231-334, 341-342, 349-351, 358-359, 362-364, 366-367, 371-1035, 1040-1051, 1055, 1063
.tox/py37/lib/python3.7/site-packages/_pytest/cacheprovider.py                            278    232    17%   7-60, 64, 67-75, 79-106, 109, 127-229, 242-253, 255-257, 261-263, 267-272, 276-332, 337, 342-349, 355, 358-427, 429-435, 441-457, 466-467, 471-508
.tox/py37/lib/python3.7/site-packages/_pytest/capture.py                                  479    407    15%   5-67, 70-133, 139, 144-153, 158-187, 192-194, 197-218, 234, 243, 245, 249, 254, 259, 264-438, 442-481, 488-491, 498-513, 517, 521, 525-584, 591-601, 606, 611-627, 635-676, 680, 684-845
.tox/py37/lib/python3.7/site-packages/_pytest/compat.py                                   178    140    21%   4-78, 83, 96-126, 158-159, 174, 185, 189-267, 283-285, 291, 295, 306, 310, 317, 327-331, 335-417, 419

[omitting 753 lines of coverage for all my site-packages]

turbot/__init__.py                                                                        661    661     0%   3-1169
turbot/migrate.py                                                                          35     35     0%   3-50
---------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                  181537 128288    29%
Coverage HTML written to dir htmlcov

========================================================================================================================================================================= 86 passed, 164 warnings in 40.09s =========================================================================================================================================================================
______________________________________________________________________________________________________________________________________________________________________________________ summary ______________________________________________________________________________________________________________________________________________________________________________________
  clean: commands succeeded
  py37: commands succeeded
  congratulations :)

Problems:

  1. Coverage was run across my all my site packages even though I have a source directive in the configuration file.
  2. The report is wrong because it says that coverage for my package is 0% which is absolutely not true.
  3. Due to running coverage against the entire world, it takes forever to finish.

I also tried using --cov=turbot and just --cov and those fail in a totally different way. It finishes instantly and also says no coverage:

---------- coverage: platform darwin, python 3.7.6-final-0 -----------
Name                 Stmts   Miss  Cover   Missing
--------------------------------------------------
turbot/__init__.py     661    661     0%   3-1169
turbot/migrate.py       35     35     0%   3-50
--------------------------------------------------
TOTAL                  696    696     0%
Coverage HTML written to dir htmlcov

Expected vs actual result

I expect to see the same results as when I run coverage directly.

Reproducer

Versions

Python 3.7.6
coverage==5.1
pytest-cov==2.8.1

Config

tox.ini

[tox]
envlist = clean,py37

[testenv]
deps = -rrequirements/dev.txt
commands =
    pytest --cov=turbot \
           --cov-report=term-missing \
           --cov-report html \
           --no-cov-on-fail

[testenv:clean]
deps = coverage
skip_install = true
commands = coverage erase

Code

All code pushed to my coverage branch at https://github.com/lexicalunit/turbot/tree/coverage

ionelmc commented 4 years ago

Well I can't access that repository. Maybe it's private?

Regardless, please see the examples here: https://github.com/pytest-dev/pytest-cov/tree/master/examples

lexicalunit commented 4 years ago

Sorry, I deleted the fork and am now just a collaborator on the source now. We've also moved away from requirements.txt and use poetry as well. Not sure why pytest-cov never worked for me, hopefully all the clues are in the description and there was nothing relevant lost when I deleted the fork.