Open DanCardin opened 2 years ago
I can confirm that there is a clear bug in this area and when I commented source
and used include
the reported coverage went from 12% to 23%, even if xdist run on 10 cores (m1).
This is still far from the real coverage on the project, which is around 84% as measured with pytest-cov.
I should mention that I reached that place while trying to avoid using pytest-cov because it was reporting some warnings about modules being imported too soon. Funny bit is that even with these warnings, the pytest-cov reported far more than coverage.
# pyproject.toml
[tool.coverage.run]
parallel = true
concurrency = ["multiprocessing", "thread"]
I am really curious to find a project using just coverage.py + pytest + xdist, what does properly record the coverage, maybe I could stop what is causing the incomplete coverage.
Another thing that I found quite weird was that I seen the coverage files being exact 102400
in file size.
After spending a good number of hours today trying to make it work w/o pytest-cov, I think I found the trick. In fact it was only one project that managed to get it working and that was via https://github.com/python-attrs/attrs/pull/1011/files change. change the the magic sauce that nobody documented was the installation of a 6 years package named coverage-enable-subprocess
-- magically once that package is installed, my coverage magically switched from 23% to 89%.
I want to understand what's going wrong here more, but:
Alternatively, if this is difficult to work around, just that perhaps the docs be made more clear that one needs to use
include
with a pattern.
The help for coverage run
says:
--include=PAT1,PAT2,...
Include only files whose paths match one of these
patterns. Accepts shell-style wildcards, which must be
quoted.
The docs for specifying source files say:
The include and omit file name patterns follow typical shell syntax: * matches any number of characters and ? matches a single character. Patterns that start with a wildcard character are used as-is, other patterns are interpreted relative to the current directory...
If you have suggestions for how to make it clearer that include
needs a pattern, I'm open to them.
I dont remember writing that 😅, the content in the docs seems clear enough. Instead it might be better to warn in the event that includes are supplied which ultimately result in no matches? But honestly that's more of an unrelated feature request.
More relevant to the bit where i said "Alternatively, if this is difficult to work around", i guess i'd now instead suggest warning if xdist is detected to be used in combination with source
; if source
cannot be made to work.
Definitely this is because you have to configure coverage.py to measure subprocesses. You can do that a number of ways:
It might be time for coverage.py to have a better way to do this itself...
This is related to #367 and #378.
Not sure if this helps, but on Windows (with coverage
, pytest
, and pytest-xdist
), if I run
coverage run --debug=trace --parallel-mode -m pytest -n auto
I consistently get an OSError: [WinError 6] The handle is invalid
. Perhaps there is a race condition when trying to write out the .coverage.*
files?
Afterwards, if I run
coverage combine && coverage report -m
I get 0%
coverage like everyone else is saying. If I don't use xdist
and just run
coverage run --debug=trace -m pytest && coverage combine && coverage report -m
Everything works fine...¯(º_o)/¯
Also unfortunately, adding coverage-enable-subprocess
to my project doesn't seem to work on my end :/
Describe the bug Either source behaves weirdly in combination with xdist, or the documentation around how source/include work could be improved.
When supplying certain source/include values in particular ways with xdist, you can either get 0% or 100%.
In my real-life usecase, I actually see 16% coverage reported, where the last xdist process reports some amount of coverage while everything else reports 0. But perhaps due to the simpler test-setup i lay out below, it only seems to result in 0s.
It seems, by description, similar to https://github.com/nedbat/coveragepy/issues/389
To Reproduce How can we reproduce the problem? Please be specific. Don't link to a failing CI job. Answer the questions below:
If you run the following, you get the correct result (86% in this case)
If you run the following, you get 0%
Perhaps notable other options I tried:
So tl;dr the behavior of source seems to be different with/without xdist, perhaps in combination given the fact that the package is nested within a src/ directory instead of
foo/
being implicitly on the path.*Expected behavior I expect the behavior of the
source
option to react the same regardless of use of the-n
flag from pytest-xdist.Alternatively, if this is difficult to work around, just that perhaps the docs be made more clear that one needs to use
include
with a pattern.