Coverage 6.0.1 reports less coverage than 5.5 for submodule

lemon24 commented 3 years ago

Describe the bug

pytest -k entry_dedupe --cov=reader.plugins.entry_dedupe reports:

100% coverage for entry_dedupe.py on Coverage 5.5
84% coverage for entry_dedupe.py on Coverage 6.0.1

pytest -k entry_dedupe --cov=reader reports 100% coverage for entry_dedupe.py regardless of version.

FWIW, the 6.0.1 HTML 84% report shows all module-level statements (imports, defs, assignments) as missing/red, but the function bodies as covered/green.

To Reproduce How can we reproduce the problem? Please be specific. Don't just link to a failing CI job. Answer the questions below:

What version of Python are you using?

CPython 3.8.1
What version of coverage.py are you using? The output of coverage debug sys is helpful.

6.0.1; coverage debug sys output:
``` -- sys ------------------------------------------------------- coverage_version: 6.0.1 coverage_module: /Users/lemon/code/reader/.venv/lib/python3.8/site-packages/coverage/__init__.py tracer: -none- CTracer: available plugins.file_tracers: -none- plugins.configurers: -none- plugins.context_switchers: -none- configs_attempted: .coveragerc setup.cfg configs_read: /Users/lemon/code/reader/setup.cfg config_file: /Users/lemon/code/reader/setup.cfg config_contents: b'[metadata]\nname = reader\nversion = attr: reader.__version__\nauthor = lemon24\ndescription = A Python feed reader library.\nlong_description = file: README.rst\nlong_description_content_type = text/x-rst\nurl = https://github.com/lemon24/reader\nlicense = BSD-3-Clause\nproject_urls =\n Documentation = https://reader.readthedocs.io/\n Code = https://github.com/lemon24/reader\n Issue tracker = https://github.com/lemon24/reader/issues\nclassifiers =\n Development Status :: 5 - Production/Stable\n Environment :: Console\n Environment :: Web Environment\n Intended Audience :: Developers\n Intended Audience :: End Users/Desktop\n License :: OSI Approved :: BSD License\n Operating System :: OS Independent\n Programming Language :: Python\n Programming Language :: Python :: 3\n Programming Language :: Python :: 3 :: Only\n Programming Language :: Python :: 3.7\n Programming Language :: Python :: 3.8\n Programming Language :: Python :: 3.9\n Programming Language :: Python :: Implementation :: CPython\n Programming Language :: Python :: Implementation :: PyPy\n Topic :: Internet\n Topic :: Internet :: WWW/HTTP :: Dynamic Content :: News/Diary\n Topic :: Internet :: WWW/HTTP :: Indexing/Search\n Topic :: Internet :: WWW/HTTP :: WSGI :: Application\n Topic :: Software Development :: Libraries\n Topic :: Utilities\n Typing :: Typed\n\nkeywords = atom, cdf, feed, rdf, rss, json feed, web feed, podcast, feed reader, feed aggregator\n\n[options]\npackages = find:\npackage_dir = = src\ninclude_package_data = true\npython_requires = >=3.7\n\ninstall_requires =\n typing-extensions\n feedparser>=6\n requests>=2.18\n # for JSON Feed date parsing\n iso8601\n\n[options.packages.find]\nwhere = src\n\n[options.extras_require]\n\n# STABLE\n\nsearch =\n beautifulsoup4>=4.5\n\n# UNSTABLE\n\ncli =\n click>=7\n # for config\n PyYAML\napp =\n flask>=0.10\n humanize\n # for config\n PyYAML\n\n# UNSTABLE PLUGINS\n\n# mushed together for convenience\nunstable-plugins =\n\n # enclosure-tags\n requests\n mutagen\n\n # preview-feed-list\n requests\n beautifulsoup4\n blinker>=1.4\n\n # sqlite-releases\n beautifulsoup4\n\n# DEVELOPMENT\n\n# run tests under one interpreter\ntests =\n pytest>=4\n pytest-randomly\n flaky\n hypothesis>=5\n coverage\n pytest-cov\n requests-mock\n # mechanicalsoup hard-depends on lxml (see below)\n mechanicalsoup; implementation_name != "pypy"\n requests-wsgi-adapter\n # we want to test search with all known bs4 parsers.\n # lxml (sometimes) fails to build on pypy, we don\'t bother.\n lxml; implementation_name != "pypy"\n html5lib\n # for _http_utils tests\n werkzeug\n # mypy does not work on pypy (yet).\n mypy; implementation_name != "pypy"\n types-requests\n\n# build docs\ndocs =\n sphinx\n sphinx_rtd_theme\n click>=7\n sphinx-click\n sphinx-issues\n sphinx-hoverxref\n sphinxcontrib-log-cabinet\n # https://github.com/readthedocs/sphinx_rtd_theme/issues/1112\n # https://github.com/readthedocs/readthedocs.org/issues/7858\n docutils==0.16\n\n# other things needed to develop / test locally / make releases\ndev =\n tox\n pre-commit\n build\n twine\n\n\n[tool:pytest]\nmarkers =\n slow: mark a test as slow.\naddopts = --no-success-flaky-report\nfilterwarnings =\n ignore:No parser was explicitly specified::reader._search\n\n\n[coverage:run]\nbranch = true\nsource =\n reader\n tests\n\n[coverage:paths]\nsource =\n src\n */site-packages\n\n\n[flake8]\n# B = bugbear\n# E = pycodestyle errors\n# F = flake8 pyflakes\n# W = pycodestyle warnings\n# B9 = bugbear opinions\nselect = B, E, F, W, B9\nignore =\n # slice notation whitespace, invalid\n E203\n # line length, handled by bugbear B950\n E501\n # bugbear line length; too sensitive, triggered for comments\n # and docstrings (and adding "noqa" in comments is making things worse);\n # black taking care of line length for code should be good enough;\n # if enabled, we should set max-line-length = 80 (so up to 88 are allowed)\n B950\n # bare except, handled by bugbear B001\n E722\n # bin op line break, invalid\n W503\n # mistakenly triggered for @overload;\n # https://github.com/PyCQA/pyflakes/issues/320 says it should be fixed\n # in 2.1.1, but it doesn\'t seem to be (it\'s fixed in master though)\n F811\nper-file-ignores =\n # __init__ modules export names\n **/__init__.py: F401\nexclude = tests/*, docs/*, scripts/*, src/reader/_vendor/*\n\n\n[mypy]\n# empty section required in 0.800; https://github.com/python/mypy/issues/9940\n[mypy-reader.__main__]\nignore_errors = True\n[mypy-reader._cli]\nignore_errors = True\n[mypy-reader._config]\nignore_errors = True\n[mypy-reader._app.*]\nignore_errors = True\n[mypy-reader._plugins.*]\nignore_errors = True\n[mypy-reader.plugins.*]\nignore_errors = True\n' data_file: -none- python: 3.8.1 (v3.8.1:1b293b6006, Dec 18 2019, 14:08:53) [Clang 6.0 (clang-600.0.57)] platform: macOS-10.15.7-x86_64-i386-64bit implementation: CPython executable: /Users/lemon/code/reader/.venv/bin/python3.8 def_encoding: utf-8 fs_encoding: utf-8 pid: 7827 cwd: /Users/lemon/code/reader path: /Users/lemon/code/reader/.venv/bin /Library/Frameworks/Python.framework/Versions/3.8/lib/python38.zip /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8 /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload /Users/lemon/code/reader/.venv/lib/python3.8/site-packages /Users/lemon/code/reader/src environment: HOME = /Users/lemon PYTEST_ADDOPTS = -p no:randomly __PYVENV_LAUNCHER__ = /Users/lemon/code/reader/.venv/bin/python3.8 command_line: /Users/lemon/code/reader/.venv/bin/coverage debug sys sqlite3_version: 2.6.0 sqlite3_sqlite_version: 3.28.0 sqlite3_temp_store: 0 sqlite3_compile_options: COMPILER=clang-6.0.0; ENABLE_FTS3_PARENTHESIS; ENABLE_FTS4 ENABLE_FTS5; ENABLE_JSON1; ENABLE_RTREE THREADSAFE=1 ```
What versions of what packages do you have installed? The output of pip freeze is helpful.

pip freeze output:
``` alabaster==0.7.12 attrs==21.2.0 Babel==2.9.1 backports.entry-points-selectable==1.1.0 beautifulsoup4==4.10.0 bleach==4.1.0 blinker==1.4 build==0.7.0 certifi==2021.5.30 cfgv==3.3.1 charset-normalizer==2.0.6 click==8.0.1 colorama==0.4.4 coverage==6.0.1 distlib==0.3.3 docutils==0.16 feedparser==6.0.8 filelock==3.3.0 flaky==3.7.0 Flask==2.0.2 html5lib==1.1 humanize==3.12.0 hypothesis==6.23.1 identify==2.3.0 idna==3.2 imagesize==1.2.0 importlib-metadata==4.8.1 iniconfig==1.1.1 iso8601==0.1.16 itsdangerous==2.0.1 Jinja2==3.0.2 keyring==23.2.1 lxml==4.6.3 MarkupSafe==2.0.1 MechanicalSoup==1.1.0 mutagen==1.45.1 mypy==0.910 mypy-extensions==0.4.3 nodeenv==1.6.0 packaging==21.0 pep517==0.11.0 pkginfo==1.7.1 platformdirs==2.4.0 pluggy==1.0.0 pre-commit==2.15.0 py==1.10.0 Pygments==2.10.0 pyparsing==2.4.7 pytest==6.2.5 pytest-cov==3.0.0 pytest-randomly==3.10.1 pytz==2021.3 PyYAML==5.4.1 -e git+https://github.com/lemon24/reader.git@5542151591d17eaa1a06da405066f679c0844553#egg=reader readme-renderer==30.0 requests==2.26.0 requests-mock==1.9.3 requests-toolbelt==0.9.1 requests-wsgi-adapter==0.4.1 rfc3986==1.5.0 sgmllib3k==1.0.0 six==1.16.0 snowballstemmer==2.1.0 sortedcontainers==2.4.0 soupsieve==2.2.1 Sphinx==4.2.0 sphinx-click==3.0.1 sphinx-hoverxref==0.7b1 sphinx-issues==1.2.0 sphinx-rtd-theme==1.0.0 sphinxcontrib-applehelp==1.0.2 sphinxcontrib-devhelp==1.0.2 sphinxcontrib-htmlhelp==2.0.0 sphinxcontrib-jsmath==1.0.1 sphinxcontrib-log-cabinet==1.0.1 sphinxcontrib-qthelp==1.0.3 sphinxcontrib-serializinghtml==1.1.5 toml==0.10.2 tomli==1.2.1 tox==3.24.4 tqdm==4.62.3 twine==3.4.2 types-requests==2.25.9 typing-extensions==3.10.0.2 urllib3==1.26.7 virtualenv==20.8.1 webencodings==0.5.1 Werkzeug==2.0.2 zipp==3.6.0 ```
What code are you running? Give us a specific commit of a specific repo that we can check out.

https://github.com/lemon24/reader/tree/5542151591d17eaa1a06da405066f679c0844553

Relevant Coverage config: https://github.com/lemon24/reader/blob/5542151591d17eaa1a06da405066f679c0844553/setup.cfg#L145-L154

To duplicate the environment, run this (in a virtualenv):
```
git clone https://github.com/lemon24/reader.git
cd reader
git reset --hard 5542151
pip install -e '.[search,cli,app,tests,docs,dev,unstable-plugins]'
```

What commands did you run?

$ coverage --version                                       
Coverage.py, version 6.0.1 with C extension
Full documentation is at https://coverage.readthedocs.io
$ rm -r .coverage                                          
$ pytest -k entry_dedupe --cov=reader.plugins.entry_dedupe 
============================= test session starts ==============================
platform darwin -- Python 3.8.1, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: /Users/lemon/code/reader, configfile: setup.cfg
plugins: flaky-3.7.0, requests-mock-1.9.3, hypothesis-6.23.1, cov-3.0.0
collected 1490 items / 1465 deselected / 25 selected                           

tests/test_plugins_entry_dedupe.py .........................             [100%]

---------- coverage: platform darwin, python 3.8.1-final-0 -----------
Name                                 Stmts   Miss Branch BrPart  Cover
----------------------------------------------------------------------
src/reader/plugins/entry_dedupe.py     134     33     73      0    84%
----------------------------------------------------------------------
TOTAL                                  134     33     73      0    84%

===================== 25 passed, 1465 deselected in 1.29s ======================

$ pytest -k entry_dedupe --cov=reader -q | grep entry_dedupe
src/reader/plugins/entry_dedupe.py            134      0     73      0   100%

Expected behavior

100% coverage for entry_dedupe.py, as Coverage 5.5 reported previously:

$ pip install coverage==5.5
$ coverage --version                                       
Coverage.py, version 5.5 with C extension
Full documentation is at https://coverage.readthedocs.io
$ rm -r .coverage                                          
$ pytest -k entry_dedupe --cov=reader.plugins.entry_dedupe 
============================= test session starts ==============================
platform darwin -- Python 3.8.1, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: /Users/lemon/code/reader, configfile: setup.cfg
plugins: flaky-3.7.0, requests-mock-1.9.3, hypothesis-6.23.1, cov-3.0.0
collected 1490 items / 1465 deselected / 25 selected                           

tests/test_plugins_entry_dedupe.py .........................             [100%]

---------- coverage: platform darwin, python 3.8.1-final-0 -----------
Name                                 Stmts   Miss Branch BrPart  Cover
----------------------------------------------------------------------
src/reader/plugins/entry_dedupe.py     134      0     73      0   100%
----------------------------------------------------------------------
TOTAL                                  134      0     73      0   100%

===================== 25 passed, 1465 deselected in 1.39s ======================

nedbat commented 3 years ago

Thanks for the good reproduction instructions! I've fixed this in 260359756694728cd13f8c8715dddf7c6e2f371d, but it means the module will be imported twice.

lemon24 commented 3 years ago

pip-install-ed a3921d2c9025f570a4c0d59d503bfd056d26e723, I'm getting 100% coverage now.

Thank you for the prompt fix!

nedbat commented 3 years ago

This is now released as part of coverage 6.0.2.

nedbat / coveragepy

Coverage 6.0.1 reports less coverage than 5.5 for submodule #1232