Debug trace option and 3rd party library detection

Describe the bug The debug trace option does not seem to report omitted files due to being detected as a 3rd party library, only as not matching the source specification. It also doesn't list files that were NOT omitted, which could also be useful. TBH, it seems like every file that gets touched at some point should be listed by "debug trace" to help with sanity checking.

To Reproduce

Use https://github.com/alexforencich/cocotbext-axi/tree/a0aad346985cc371dd63b0b2880e7850a7b0ef29
Edit setup.cfg to remove usedevelop=True and add coverage option debug=trace under new section [coverage:run]
Run tox -- --verbose -k test_axis -n 3 (may also need to install icarus verilog)
Note that in the combined coverage report, most files have no coverage info, and no files are reported as being excluded for being 3rd party library code

Expected behavior The python code that tox installed should be reported as being excluded for some reason, since it seems to be getting excluded. Contrast with running with usedevelop=True, where these files are not excluded.

Additional context Is there a way for coverage to handle tox-installed python modules properly, or is usedevelop the correct solution?

Oh, one other thing that I just tried; I added --cov={envsitepackagesdir}/cocotbext/axi to the pytest command and removed usedevelop. The result is that the files are excluded, and nothing is printed about the files being excluded. However, it does report "Source is in third-party because of source directory '/home/alex/Projects/cocotbext-axi/.tox/py38/lib/python3.8/site-packages/cocotbext/axi'". So, it seems like what might be happening is that the 3rd-party library matches are getting removed before the sources checks (and debug printing) is being done.

I'd like to try reproducing this, but it would be easier if you could create a branch that has all of the edits needed. Also, can you try this without pytest-cov? It's possible that some of the debugging output is being captured by pytest. One last thing to try: set COVERAGE_DEBUG_FILE to a file name to get the debug output independent of any test runner interactions.

Another thing that can help my debugging: what's a specific file that is being handled incorrectly?

I tried your code. When I removed usedevelop, and wrote the debug output to a file, I see this for axis.py (a 0% file I picked at random):

Not tracing '/here/.tox/py37/lib/python3.7/site-packages/cocotbext/axi/axis.py': falls outside the --source spec

Is that the sort of trace output you were looking for?

Yes, but that's not in the output that I am getting. Here is the output for the python 3.7 run only:

$ tox -- --verbose -k test_axis -n 3
GLOB sdist-make: /home/alex/Projects/cocotbext-axi/setup.py
py37 recreate: /home/alex/Projects/cocotbext-axi/.tox/py37
py37 installdeps: pytest == 7.2.1, pytest-xdist == 3.1.0, cocotb == 1.6.2, cocotb-bus == 0.2.1, cocotb-test == 0.2.2, coverage == 7.0.5, pytest-cov == 4.0.0
py37 inst: /home/alex/Projects/cocotbext-axi/.tox/.tmp/package/1/cocotbext-axi-0.1.19.zip
py37 installed: attrs==22.2.0,cocotb==1.6.2,cocotb-bus==0.2.1,cocotb-test==0.2.2,cocotbext-axi @ file:///home/alex/Projects/cocotbext-axi/.tox/.tmp/package/1/cocotbext-axi-0.1.19.zip,coverage==7.0.5,exceptiongroup==1.1.0,execnet==1.9.0,importlib-metadata==6.0.0,iniconfig==2.0.0,packaging==23.0,pluggy==1.0.0,pytest==7.2.1,pytest-cov==4.0.0,pytest-xdist==3.1.0,tomli==2.0.1,typing_extensions==4.4.0,zipp==3.11.0
py37 run-test-pre: PYTHONHASHSEED='319103685'
py37 run-test: commands[0] | pytest --cov=cocotbext --cov=tests --cov-branch --verbose -k test_axis -n 3
======================================================================================== test session starts ========================================================================================
platform linux -- Python 3.7.15, pytest-7.2.1, pluggy-1.0.0 -- /home/alex/Projects/cocotbext-axi/.tox/py37/bin/python
cachedir: .tox/py37/.pytest_cache
rootdir: /home/alex/Projects/cocotbext-axi, configfile: setup.cfg, testpaths: tests
plugins: cov-4.0.0, cocotb-test-0.2.2, xdist-3.1.0
[gw0] linux Python 3.7.15 cwd: /home/alex/Projects/cocotbext-axi
[gw1] linux Python 3.7.15 cwd: /home/alex/Projects/cocotbext-axi
[gw2] linux Python 3.7.15 cwd: /home/alex/Projects/cocotbext-axi
[gw0] Python 3.7.15 (default, Nov 22 2022, 22:43:41)  -- [GCC 12.2.0]
[gw1] Python 3.7.15 (default, Nov 22 2022, 22:43:41)  -- [GCC 12.2.0]
[gw2] Python 3.7.15 (default, Nov 22 2022, 22:43:41)  -- [GCC 12.2.0]
gw0 [3] / gw1 [3] / gw2 [3]
scheduling tests via LoadScheduling

tests/axis/test_axis.py::test_axis[8] 
tests/axis/test_axis.py::test_axis[32] 
tests/axis/test_axis.py::test_axis[16] 
[gw2] [ 33%] PASSED tests/axis/test_axis.py::test_axis[32] Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/pytest_cov/compat.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/coverage/control.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/coverage/collector.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/assertion/__init__.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/runner.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/reports.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/doctest.py': falls outside the --source spec

[gw1] [ 66%] PASSED tests/axis/test_axis.py::test_axis[16] Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/pytest_cov/compat.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/coverage/control.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/coverage/collector.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/assertion/__init__.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/runner.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/reports.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/doctest.py': falls outside the --source spec

[gw0] [100%] PASSED tests/axis/test_axis.py::test_axis[8] Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/pytest_cov/compat.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/coverage/control.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/coverage/collector.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/assertion/__init__.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/runner.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/reports.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/doctest.py': falls outside the --source spec

---------- coverage: platform linux, python 3.7.15-final-0 -----------
Name                               Stmts   Miss Branch BrPart  Cover
--------------------------------------------------------------------
cocotbext/axi/__init__.py             16     16      0      0     0%
cocotbext/axi/address_space.py       260    260     62      0     0%
cocotbext/axi/axi_channels.py         53     53      0      0     0%
cocotbext/axi/axi_master.py          714    714    252      0     0%
cocotbext/axi/axi_ram.py              19     19      0      0     0%
cocotbext/axi/axi_slave.py           223    223     72      0     0%
cocotbext/axi/axil_channels.py        53     53      0      0     0%
cocotbext/axi/axil_master.py         396    396    126      0     0%
cocotbext/axi/axil_ram.py             19     19      0      0     0%
cocotbext/axi/axil_slave.py          164    164     48      0     0%
cocotbext/axi/axis.py                569    569    293      0     0%
cocotbext/axi/buddy_allocator.py      44     44     20      0     0%
cocotbext/axi/constants.py            55     55      0      0     0%
cocotbext/axi/memory.py               57     57      6      0     0%
cocotbext/axi/reset.py                32     32      8      0     0%
cocotbext/axi/sparse_memory.py        67     67     32      0     0%
cocotbext/axi/stream.py              278    278    106      0     0%
cocotbext/axi/utils.py                19     19      6      0     0%
cocotbext/axi/version.py               1      1      0      0     0%
tests/axi/test_axi.py                221    196     64      1     9%
tests/axil/test_axil.py              203    178     60      1    10%
tests/axis/test_axis.py               98     62     13      1    35%
tests/test_buddy_allocator.py         14     12      4      0    11%
tests/test_sparse_memory.py           29     27     10      0     5%
--------------------------------------------------------------------
TOTAL                               3604   3514   1182      3     2%

========================================================================================= 3 passed in 6.26s =========================================================================================
sys.path:
    /home/alex/Projects/cocotbext-axi/.tox/py37/bin
    /usr/lib/python37.zip
    /usr/lib/python3.7
    /usr/lib/python3.7/lib-dynload
    /home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages
Source matching against trees <TreeMatcher source ['/home/alex/Projects/cocotbext-axi/cocotbext', '/home/alex/Projects/cocotbext-axi/tests']>
Coverage code matching: <TreeMatcher coverage ['/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/coverage']>
Third-party lib matching: <TreeMatcher third ['/home/alex/.local/bin', '/home/alex/.local/lib/python3.7/site-packages', '/home/alex/Projects/cocotbext-axi/.tox/py37/bin', '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python', '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages']>
Not tracing '/usr/lib/python3.7/threading.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/config/__init__.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/pluggy/_manager.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/inspect.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/pluggy/_hooks.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/enum.py': falls outside the --source spec
Not tracing '<string>': not a real original file name
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/pluggy/_callers.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/legacypath.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/pluggy/_result.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/python_path.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/monkeypatch.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/pathlib.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/pathlib.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/posixpath.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/_collections_abc.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/fnmatch.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/re.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/sre_compile.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/sre_parse.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/capture.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/warnings.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/contextlib.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/warnings.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/config/compat.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/config/argparsing.py': falls outside the --source spec
Not tracing '<frozen importlib._bootstrap>': not a real original file name
Not tracing '/usr/lib/python3.7/argparse.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/gettext.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/os.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/locale.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/encodings/__init__.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/genericpath.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/terminal.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/helpconfig.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/_io/terminalwriter.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/shutil.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/types.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/_argcomplete.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/xdist/plugin.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/typing.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/shlex.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/glob.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/setupplan.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/setuponly.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/mark/__init__.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/xdist/looponfail.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/cacheprovider.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/python.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/main.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/nodes.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/mark/structures.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/stash.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/pluggy/_tracing.py': falls outside the --source spec
Not tracing '<attrs generated init _pytest.main._bestrelpath_cache>': not a real original file name
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/deprecated.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/cocotb_test/plugin.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/pytest_cov/plugin.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/faulthandler.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/assertion/rewrite.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_distutils_hack/__init__.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_virtualenv.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/codecs.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/stepwise.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/junitxml.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/tmpdir.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/skipping.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/debugging.py': falls outside the --source spec
Not tracing '<frozen importlib._bootstrap_external>': not a real original file name
Not tracing '/usr/lib/python3.7/pdb.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/cmd.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/code.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/codeop.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/pastebin.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/logging.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/logging/__init__.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/_weakrefset.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/ast.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/xdist/dsession.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/xdist/remote.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/importlib_metadata/_compat.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/xdist/workermanage.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/typing_extensions.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/xdist/scheduler/__init__.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/xdist/scheduler/each.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/xdist/report.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/xdist/scheduler/load.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/xdist/scheduler/loadfile.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/xdist/scheduler/loadscope.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/xdist/scheduler/loadgroup.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/queue.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/fixtures.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/compat.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/_py/path.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/scope.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/pytest/__init__.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/_io/wcwidth.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/runner.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/platform.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/functools.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/importlib_metadata/__init__.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/email/__init__.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/email/parser.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/email/feedparser.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/email/message.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/email/_policybase.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/importlib_metadata/_adapters.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/email/utils.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/uuid.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/execnet/multi.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/execnet/gateway_base.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/execnet/xspec.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/execnet/gateway_io.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/subprocess.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/execnet/gateway_bootstrap.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/execnet/gateway.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/weakref.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/linecache.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/tokenize.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/textwrap.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/tempfile.py': falls outside the --source spec
Not tracing '/usr/lib/python3.7/getpass.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/pytest_cov/engine.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/_pytest/reports.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/pytest_cov/compat.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/coverage/control.py': falls outside the --source spec
Not tracing '/home/alex/Projects/cocotbext-axi/.tox/py37/lib/python3.7/site-packages/coverage/collector.py': falls outside the --source spec
py37 run-test: commands[1] | bash -c 'find . -type f -name "\.coverage" | xargs coverage combine --append'
Combined data file tests/axis/sim_build/test_axis-16/.coverage
Skipping duplicate data tests/axis/sim_build/test_axis-8/.coverage
Skipping duplicate data tests/axis/sim_build/test_axis-32/.coverage
py37 run-test: commands[2] | coverage report
Name                               Stmts   Miss Branch BrPart  Cover
--------------------------------------------------------------------
cocotbext/axi/__init__.py             16     16      0      0     0%
cocotbext/axi/address_space.py       260    260     62      0     0%
cocotbext/axi/axi_channels.py         53     53      0      0     0%
cocotbext/axi/axi_master.py          714    714    252      0     0%
cocotbext/axi/axi_ram.py              19     19      0      0     0%
cocotbext/axi/axi_slave.py           223    223     72      0     0%
cocotbext/axi/axil_channels.py        53     53      0      0     0%
cocotbext/axi/axil_master.py         396    396    126      0     0%
cocotbext/axi/axil_ram.py             19     19      0      0     0%
cocotbext/axi/axil_slave.py          164    164     48      0     0%
cocotbext/axi/axis.py                569    569    293      0     0%
cocotbext/axi/buddy_allocator.py      44     44     20      0     0%
cocotbext/axi/constants.py            55     55      0      0     0%
cocotbext/axi/memory.py               57     57      6      0     0%
cocotbext/axi/reset.py                32     32      8      0     0%
cocotbext/axi/sparse_memory.py        67     67     32      0     0%
cocotbext/axi/stream.py              278    278    106      0     0%
cocotbext/axi/utils.py                19     19      6      0     0%
cocotbext/axi/version.py               1      1      0      0     0%
tests/axi/test_axi.py                221    196     64      1     9%
tests/axil/test_axil.py              203    178     60      1    10%
tests/axis/test_axis.py               98      0     13      0   100%
tests/test_buddy_allocator.py         14     12      4      0    11%
tests/test_sparse_memory.py           29     27     10      0     5%
--------------------------------------------------------------------
TOTAL                               3604   3452   1182      2     3%

No matches at all for site-packages/cocotbext in that listing.

So, I just tried with COVERAGE_DEBUG_FILE set, and there are entries for the files in question in the file that was written out. Interesting, so maybe this is just a problem with pytest-cov or pytest screwing with the output of coverage. It's interesting that there seems to be a random snippet of debug output that does get displayed, hence I thought the problem was with coverage not printing out debug information correctly instead of some other entity throwing away output arbitrarily.

It's also interesting that this debug output lists the files as not being in the source spec. The incorrect coverage data started after version 5.5 (coverage version 5.5 works, 6.0 and newer does not) and the only significant change that seems like it could contribute is the 3rd party library detection that was added in 6.0, so my assumption is that it is this 3rd party code detection that's the cause of the problem. But perhaps something else is going on. At any rate, debugging this breakage was not straightforward; I'm not sure what could be done to improve the situation.

Also, I was reading that coverage should work for installed modules if the module name is specified in --cov. So, I just tried adding --cov=cocotbext.axi to the pytest command in tox.ini and running it again, also with COVERAGE_DEBUG_FILE set.

Observations:

Coverage does not trace the extension's files (most files have 0% coverage in the final combined coverage report)
Coverage detects the cocotbext.axi source package, and determines that it is "in third-party"
SOME of the coverage runs in the log file mention tracing the extension files, but most have no mention of the files at all

So, it seems like there may still be something fishy going on here. I can post the log file (~6k lines) if you can't reproduce this on your end.

I added more documentation, and the possibility of setting the debug file in the settings, in commit 5f65d87b14245d4523bc866a75a16b6c55a7ce70.

@alexforencich:

I don't know what you mean by "the extension". Is this entire package "the extension"?
Part of the problem here is that import cocotbext will find your local source directory, instead of the package installed in site-packages. When you tell coverage what to watch, it is looking in the wrong place. A solution to this is to use a "src layout" instead.

By "the extension" I'm referring to all of the python code in "cocotbext/axi" (as it is a "cocotb extension") which is importable as cocotbext.axi, as opposed to the test code in the tests directory, which always reports nonzero coverage. "cocotbext" is also a namespace package; I'm not sure if that has any relevance here.

Also, the whole process of running the code that needs to be profiled is a bit complex. Basically it works like this: pytest runs the testbenches in the test directory. These testbenches use cocotb-test to run icarus verilog, which internally spins up another python interpreter which then pulls in the extension code and runs a set of "inner" tests with cocotb's internal test runner. The python code running inside icarus verilog is what really needs to be profiled, and since that interpreter is started inside the tests directory, it can't see the "cocotbext" directory in the root, it can only see the installed package that's installed by tox, be it an editable package that refers to the "local" source code, or a normal package that's installed in site-packages in the venv. Well, unless perhaps there is something screwy with the python search path or something that I'm not aware of. So I'm not sure if using an "src layout" would make much difference in this case, although I suppose it wouldn't be too hard to test that. And coverage does seem to be detecting the installed package as it reports that "Source is in third-party" when I add --cov=cocotbext.axi or --cov={envsitepackagesdir}/cocotbext/axi.

Anyway, to summarize the behavior I have observed so far:

coverage 5.5, no usedevelop: works
coverage 6.0 and 7.0.5, with usedevelop: works
coverage 7.0.5, no usedevelop: many files report zero coverage, debug log shows extension files excluded due to --source spec
coverage 7.0.5, no usedevelop, added --cov=cocotbext.axi: many files report zero coverage, extension files appear in log either as "tracing" or they do not appear at all
coverage 7.0.5, no usedevelop, added --cov={envsitepackagesdir}/cocotbext/axi: many files report zero coverage, extension files appear in log either as "tracing" or they do not appear at all
coverage 7.0.5, no usedevelop, replaced --cov=cocotbext with --cov=cocotbext.axi: many files completely missing from coverage report, extension files appear in log either as "tracing" or they do not appear at all

At no point have I seen a debug log indicating one of the extension files being excluded for any reason other than "falls outside the --source spec", so this leads to my hypothesis that maybe there is some sort of filtering, possibly related to third-party code, that is taking place before the debug code is run, and as a result there is no log output referencing these files.

Also, I'm wondering if it might be a good idea to add some higher-level debug information (for example, process, command line, etc.) to the debug output. When there are multiple processes involved, each one dumps a bunch of lines into the shared coverage debug log, and it's not necessarily clear which block of output came from which process.

I also did some screwing around with path mappings, but didn't get anywhere. I also added --keep to the combine run and looked at the intermediate coverage results, but they only included the test code, not the extension code, at least for the cases where the coverage was incorrect.

which internally spins up another python interpreter which then pulls in the extension code

You might need to enable subprocess support: https://coverage.readthedocs.io/en/7.0.5/subprocess.html

coverage 6.0 and 7.0.5, with usedevelop: works

Is there a reason you can't continue to use usedevelop?

Also, I'm wondering if it might be a good idea to add some higher-level debug information (for example, process, command line, etc.) to the debug output. When there are multiple processes involved, each one dumps a bunch of lines into the shared coverage debug log, and it's not necessarily clear which block of output came from which process.

Take a look at the debug options available: you probably want process and pid in addition to trace.

Seems like without some kind of subprocess support already working, coverage would not work at all. Cocotb has its own provisions for running coverage by setting the COVERAGE environment variable. But, I suppose it's possible that cocotb is doing something strange with the configuration of coverage when it starts it up inside of the simulator (in this case, icarus verilog), and that may be the cause of the seemingly strange behavior here.

After adding the process and pid debug options, it looks like there isn't any debug output from coverage running inside icarus verilog. Every unique PID that I can see lists python as the executable when it starts writing debug information, and I only see 6 different processes for each test env - with xdist running 3 processes in parallel and each one of those running the simulator plus the original pytest process, I would expect there to be more than 6 processes. There is also quite a bit of interleaving between the processes and each process seems to dump the config twice, which clears things up a bit - it seems that all of the sub-processes started by xdist are actually listing the extension files as being traced. I also did a quick test by setting the COVERAGE_DEBUG env var instead of using the [coverage:run] section in setup.cfg, but got the same result (no debug output from coverage running in iverilog).

So perhaps the problem in this case is that the config options are not being fully passed across to the coverage instances started by cocotb, so all of the messing around with the path settings and module targets and such aren't actually affecting the coverage instances running inside of iverilog that actually matter.

Incidentally, it might be useful to include the pid in the name of the debug output file, so that when multiple processes are involved, separate per-process logs are written instead of one log file that's subject to a lot of interleaving.

There isn't really a reason I can't use usedevelop, it's just that the behavior didn't make sense and I suspected there may be some sort of a bug somewhere that should be isolated and fixed. And it also worked without usedevelop with coverage version 5.5. If usedevelop is the only way to make everything work (and there aren't any other bugs to be resolved), then I will just use that going forward.

It sounds to me like you need to dig into what cocotb is doing with coverage. I don't know what cocotb's COVERAGE environment variable does.

Looks like cocotb supports two env vars - COVERAGE and COCOTB_LIBRARY_COVERAGE. The first one is for coverage of extensions and testbench code, the second is for coverage of cocotb itself. In this case I am only using the first one.

Not much takes place for the first one, looks like there are only two places in the code that are relevant:

Conditional import of coverage: https://github.com/cocotb/cocotb/blob/071f6e20812670cff489b99eea349bd20239808a/cocotb/regression.py#L58-L68
Call to coverage.coverage() to start tracing: https://github.com/cocotb/cocotb/blob/071f6e20812670cff489b99eea349bd20239808a/cocotb/regression.py#L119-L134

Looks like only branch and omit are specified in the call to coverage.coverage(), presumably everything else should come from the config file.

But, what I noticed is that I am specifying the sources on the command line to pytest-cov via --cov parameters....it seems like there is no way for this information to get from pytest-cov into cocotb. So, I moved the sources into setup.cfg and removed the --cov parameters. This still did not work...then I figured out another problem: the "inner" coverage is being run with a working directory down inside the test directories, so as a result it is actually writing out debug info, just in a different file, and after looking at that file I noticed that it is not locating the setup.cfg file that has all of the configuration as it seems coverage doesn't go up any levels in search of a config file. So, I also explicitly set the env var COVERAGE_RCFILE={toxinidir}/setup.cfg. This seemed to finally get the .coverage files in the test directories to contain the coverage data. One final change was needed to get things to combine correctly, a path entry of .tox/*/lib/python*/site-packages/cocotbext/.

So, it seems like there are no major bugs, just mainly a poor hand-off between pytest-cov and cocotb in terms of the coverage configuration.

I suppose there are maybe a couple of things that could be improved upon here:

Option to put the current PID in the debug file name, to aid in debugging when multiple processes are involved. Maybe something similar to the current parallel option for the coverage data files.
Maybe consider searching for config files in parents of the working directory, to better support running from a sub-directory
Maybe have an option to specify the root for path matching. It seems that paths are relative to the current working directory. While tox has options like {toxinidir} that tox can substitute for the repo root directory, these can't be used in non-tox sections in tox.ini since this will be read directly by other tools, so there doesn't seem to be a good way to specify absolute paths. From looking at the debug output, the path mapping only works during the final combine step, as the relative path doesn't match from within the individual test directories. I also look a look at the relative_path config option, but can't figure out how it's supposed to work.

I'm thinking about how to potentially improve the hand-off between pytest and cocotb. Seems like cocotb-test is probably the best place for something like this, since cocotb-test gets called by pytest and is responsible for starting the simulator process, so it would be a good place to do things like setting env vars when starting the simulator.

In this case, cocotb already has provisions for running coverage, so I think we can just use that instead of setting COVERAGE_PROCESS_START and modifying the python installation. Possibly setting COVERAGE_RCFILE automatically based on what pytest-cov is doing might make sense, but I'm not sure if this would be a complete solution since coverage is still running in a different working directory and as a result things like path matching don't work quite right.

Actually, never mind, even getting the "inner" coverage instance to look at the correct config file does not work properly, as in that case the test code itself is excluded. The issue in this case is that the test code is, for instance, in tests/axis/, while the working directory when running coverage is set to tests/axis/sim_build/test_axis-16. So for the "outer" coverage, specifying "tests" in sources works, but this doesn't work for the "inner" coverage. It seems that there are some significant shortcomings in terms of how coverage handles paths, and short of writing out config files for each test, either manually or through some automated process, it's not clear how to make all of this work, aside from simply using usedevelop and relying on the default 3rd party code detection and the final merge and path remapping step to do the right thing.

nedbat / coveragepy

Debug trace option and 3rd party library detection #1540