Open nedbat opened 7 years ago
Thanks for the detailed instructions. I had to install a bunch of requirements (including enum34 manually, which isn't mentioned in any requirements file?). But then I did not see the results you see:
$ PYTHONPATH=src python -m coverage run --branch --rcfile=/dev/null --include=src/hypothesis/searchstrategy/strategies.py test_local.py
$ coverage report --rcfile=/dev/null --show-missing
Name Stmts Miss Branch BrPart Cover Missing
-----------------------------------------------------------------------------------------
src/hypothesis/searchstrategy/strategies.py 244 116 68 3 46% 37-40, 143, 170-173, 212, 225-267, 278, 290-291, 304, 311, 320-322, 330-339, 342, 345, 370-371, 384-402, 405-414, 417, 420-421, 425-432, 445-448, 451, 454-459, 462, 467, 471-478, 482, 491-493, 496, 499-504, 507, 510-526, 530-534, 133->143, 369->370, 377->exit
$ PYTHONPATH=src python -m coverage run --rcfile=/dev/null --include=src/hypothesis/searchstrategy/strategies.py test_local.py
$ coverage report --rcfile=/dev/null --show-missing
Name Stmts Miss Cover Missing
---------------------------------------------------------------------------
src/hypothesis/searchstrategy/strategies.py 244 116 52% 37-40, 143, 170-173, 212, 225-267, 278, 290-291, 304, 311, 320-322, 330-339, 342, 345, 370-371, 384-402, 405-414, 417, 420-421, 425-432, 445-448, 451, 454-459, 462, 467, 471-478, 482, 491-493, 496, 499-504, 507, 510-526, 530-534
Hi,
I was able to get probably the same bug with a minimal example with pytest:
test_foo.py
def test_literal_list():
assert(
[0,1,2] == [0,1,2]
)
def test_foobar():
assert(
[0,1,2] == list(x for x in range(3))
)
def test_foobar2():
assert (
[x for x in range(3)] == [0, 1, 2]
)
def test_foobar2_oneline():
assert ([x for x in range(3)] == [0, 1, 2])
pytest --cov-branch --cov-report term-missing
(which more or less translates to coverage report -m
with branch coverage)
Results:
----------- coverage: platform linux, python 3.5.2-final-0 -----------
Name Stmts Miss Branch BrPart Cover Missing
-----------------------------------------------------------------------
test_foo.py 8 0 6 2 86% 7->exit, 12->exit
As you can see, the first test was "covered" in full. The test_foobar
and test_foobar2
seem to be covered partially, and test_foobar2_oneline
which is the same as test_foobar2
, but without splitting this into multiple lines, is "covered" in full.
Also, I made another test - the same file with executing the test functions manually and without pytest at all:
def test_literal_list():
assert(
[0,1,2] == [0,1,2]
)
def test_foobar():
assert(
[0,1,2] == list(x for x in range(3))
)
def test_foobar2():
assert (
[x for x in range(3)] == [0, 1, 2]
)
def test_foobar2_oneline():
assert ([x for x in range(3)] == [0, 1, 2])
test_literal_list()
test_foobar()
test_foobar2()
test_foobar2_oneline()
coverage run --branch test_foo.py
coverage report -m
Name Stmts Miss Branch BrPart Cover Missing
---------------------------------------------------------
test_foo.py 12 0 6 0 100%
Note that:
Coverage 4.5.1 (I had the same issue with 4.0, but with different result presentation 7->-8
instead of 7->exit
)
pytest 3.5.0 pytest-cov 2.5.1
@bluefish6 We should probably create this as a new bug, in case it is not the same cause as the original report here.
But I tried running your code, and did not see the same results. pytest-cov didn't produce any reporting output:
$ pytest --cov-branch --cov-report term-missing
=================================== test session starts ====================================
platform darwin -- Python 3.6.6, pytest-3.5.0, py-1.5.4, pluggy-0.6.0
rootdir: /Users/ned/coverage/bug605, inifile:
plugins: cov-2.5.1
collected 4 items
test_foo.py .... [100%]
================================= 4 passed in 0.01 seconds =================================
When I run with coverage directly, I get 100% coverage:
$ coverage run --source=. -m py.test
=================================== test session starts ====================================
platform darwin -- Python 3.6.6, pytest-3.5.0, py-1.5.4, pluggy-0.6.0
rootdir: /Users/ned/coverage/bug605, inifile:
plugins: cov-2.5.1
collected 4 items
test_foo.py .... [100%]
================================= 4 passed in 0.01 seconds =================================
$ coverage report -m
Name Stmts Miss Cover Missing
-------------------------------------------
test_foo.py 8 0 100%
The code I ran:
$ cat test_foo.py
def test_literal_list():
assert(
[0,1,2] == [0,1,2]
)
def test_foobar():
assert(
[0,1,2] == list(x for x in range(3))
)
def test_foobar2():
assert (
[x for x in range(3)] == [0, 1, 2]
)
def test_foobar2_oneline():
assert ([x for x in range(3)] == [0, 1, 2])
$ python -V
Python 3.6.6
$ pip freeze
atomicwrites==1.1.5
attrs==18.1.0
coverage==4.5.1
more-itertools==4.2.0
pluggy==0.6.0
py==1.5.4
pytest==3.5.0
pytest-cov==2.5.1
six==1.11.0
Hi,
I was seeing similar behavior where lambdas being passed to a method were being reported as a missed arc in python 2.7.6, python 3.5.1 and python 3.6.0 (if that matters). In my case, changing the lambdas to the appropriate partial caused the coverage check to pass. Interestingly, in a chain of method calls in which two lambdas are passed, the same line (ie 47->exit, 47->exit
) is reported twice.
I don't have time right now but I'll take a closer look later today.
@nedbat Hi Ned,
Thanks for taking a look!
I'm terribly sorry - I mistakenly did not include the parameter --cov=.
.
It should be:
py.test --cov=. --cov-report term-missing --cov-branch
================================== test session starts ==================================
platform linux -- Python 3.5.2, pytest-3.6.3, py-1.5.4, pluggy-0.6.0
rootdir: /home/antek/praca/evox/coverage_test, inifile:
plugins: cov-2.5.1
collected 4 items
test_foo.py .... [100%]
----------- coverage: platform linux, python 3.5.2-final-0 -----------
Name Stmts Miss Branch BrPart Cover Missing
---------------------------------------------------------
test_foo.py 12 0 6 2 89% 7->exit, 12->exit
=============================== 4 passed in 0.03 seconds ================================
I was able to reproduce it also with:
coverage run --source=. -m py.test --cov=. --cov-branch --cov-report term-missing
================================== test session starts ==================================
platform linux -- Python 3.5.2, pytest-3.6.3, py-1.5.4, pluggy-0.6.0
rootdir: /home/antek/praca/evox/coverage_test, inifile:
plugins: cov-2.5.1
collected 4 items
test_foo.py .... [100%]
----------- coverage: platform linux, python 3.5.2-final-0 -----------
Name Stmts Miss Branch BrPart Cover Missing
---------------------------------------------------------
test_foo.py 12 0 6 2 89% 7->exit, 12->exit
=============================== 4 passed in 0.04 seconds ================================
Coverage.py warning: No data was collected. (no-data-collected)
I solved the mystery. The one to blame is pytest and it's "assertion rewrite hook". As far as I understand, it changes the code of all assert
s to be a special fancy assert-like function that shows better errors. The problem is, that for some reason it breaks the coverage for cases like:
assert (
[x for x in range(3)] == [0, 1, 2]
)
Quick test - let's pretend that our system does not support the hook: (.../site-packages/_pytest/assertion/__init.__.py
) and always raise the Jython-dependent error:
def install_importhook(config):
"""Try to install the rewrite hook, raise SystemError if it fails."""
raise SystemError("rewrite not supported") # <-- add this to force an Exception to not install the hook
# Jython has an AST bug that make the assertion rewriting hook malfunction.
if sys.platform.startswith("java"):
raise SystemError("rewrite not supported")
And voilla, no branches uncovered.
So to sum up - the bug is somewhere inside the assertion rewrite mechanism of pytest. I will open an issue there with link to this issue for reference.
Issue created in pytest repo: https://github.com/pytest-dev/pytest/issues/3689
Possibly my bug is a duplicate of https://github.com/nedbat/coveragepy/issues/515
Sadly this 5 year old bug is still valid and probably it would worth even a runtime warning for those that make the mistake of enabling branch coverage.
It has nothing to do with pytest, as I had the code below that also reports no branch coverage, even if I tested with the debugger to see that the loop really executes.
if isinstance(when, list):
for item in when:
return _changed_in_when(item)
@ssbarnea can you provide a complete example?
I will to isolate it after I deal with current pile of incoming changes. You can see that mentioned disablement that increases total coverage ~4% by disabling branch coverage. I know that branch is good, but is good only when it works.
I wonder how hard it will be to reproduce it in isolation, as we do use xdist and subprocess coverage commands too, I would not be surprised to discover that the issue happens only in conjunction with one of these.
@ssbarnea I saw the same issue with branch, using next("comprehension with if", None)
. Turns out coverage
was correct: i wasn't testing the None
case.
In your case, you may be assuming _changed_in_when
is called, but you should consider the possibility it isn't (namely, if when
is an empty iterable).
Originally reported by David MacIver (Bitbucket: davidmaciver, GitHub: Unknown)
I ran into this problem when testing Hypothesis: Lines that are definitely covered by a test are showing up as uncovered in coverage.
The following is the test file:
When testing Hypothesis commit 8844ca758737a6439a82507e7910972c6fa22358 (bb), this shows lines 156-158 in src/hypothesis/searchstrategy/strategies.py as uncovered. However the lines are definitely covered by this test (see below for evidence).
To reproduce this problem:
(now copy the above example as test_local.py)
Salient features: