coverage is wrong when running with xdist's --boxed

diefans commented 10 years ago

I need to run my tests with xdist's --boxed feature. there is a big difference in coverage compared to running unboxed:

unboxed: 93.48 %
boxed: 26.93 %

35 % of all modules are completely uncovered. I also noticed, that despite that I run the tests with py.test tests/ --boxed -d -n 8 --random the uncovered modules are always the same.

When I run one single test module in boxed mode coverage complains that the module to cover was not imported:

py.test tests/unit/security/test_features.py --boxed -d -n 8 --cov bm.security
===================================================================================================== test session starts =====================================================================================================
platform linux2 -- Python 2.7.5 -- pytest-2.5.0
Tests are shuffled using seed number 358684703390.
plugins: random, bdd, cov, capturelog, ipdb, cache, pep8, greendots, xdist
gw0 [49] / gw1 [49] / gw2 [49] / gw3 [49] / gw4 [49] / gw5 [49] / gw6 [49] / gw7 [49]
scheduling tests via LoadScheduling
................................s..Coverage.py warning: Module bm.security was never imported.
Coverage.py warning: No data was collected.
.......Coverage.py warning: Module bm.security was never imported.
Coverage.py warning: No data was collected.
.Coverage.py warning: Module bm.security was never imported.
Coverage.py warning: No data was collected.
.Coverage.py warning: Module bm.security was never imported.
Coverage.py warning: No data was collected.
.Coverage.py warning: Module bm.security was never imported.
Coverage.py warning: No data was collected.
.Coverage.py warning: Module bm.security was never imported.
Coverage.py warning: No data was collected.
.Coverage.py warning: Module bm.security was never imported.
Coverage.py warning: No data was collected.
.Coverage.py warning: Module bm.security was never imported.
Coverage.py warning: No data was collected.
.
--------------------------------------------------------------------------------------- coverage: platform linux2, python 2.7.5-final-0 ---------------------------------------------------------------------------------------
Name    Stmts   Miss     Cover   Missing
----------------------------------------
============================================================================================ 48 passed, 1 skipped in 3.60 seconds =============================================================================================

schlamar commented 10 years ago

Can you please test against latest master, there are various improvements/fixes not yet released.

diefans commented 10 years ago

Sorry, I get the same results...

mattupstate commented 10 years ago

I was experiencing a similar issue where running:

$ py.test --cov mymodule --cov-report=term-missing

was reporting correct coverage. Then when running through setuptools like:

$ python setup.py test

The coverage was incorrect. However, installing the latest from master branch has fixed my issue.

schlamar commented 10 years ago

@diefans Confirmed: https://travis-ci.org/schlamar/pytest-cov/builds/26548730

@mattupstate That seems unrelated, but thanks for confirming that master solves a few issues.

schlamar commented 10 years ago

This is an upstream issue: https://bitbucket.org/hpk42/py/issue/45/forkedfunc-skips-exit-handlers-breaks

schlamar commented 10 years ago

@diefans I have a hacky but working solution by patching py.process.ForkedFunc. Not sure if I want to roll that out officially. But it should work if you add this to your conftest.py: https://github.com/schlamar/pytest-cov/commit/fb100dba36079227a3f491d3d055f9e377031c59

schlamar commented 10 years ago

I have prepared a much nicer solution upstream:

py pull request
fix in cov-core https://github.com/schlamar/cov-core/commit/52086bc91b4cc65bca2b8714f5e957042e592fe3

sontek commented 10 years ago

@schlamar Since this is upstream bug, does this mean you can release without the monkeypatching? Or are you going to wait for upstream anyways?

schlamar commented 10 years ago

@sontek I'm going to wait for the decision on my pull request for py. If it is merged I can release because pytest-cov doesn't need any further changes.

schlamar commented 10 years ago

@sontek 1.7.0 just released

@diefans this will be fixed as soon as a new py version is released, my pull request was merged

schlamar commented 10 years ago

@diefans should be fixed, you just need to upgrade py to 1.4.21 and cov-core to 1.13.0.

mocksoul commented 10 years ago

Broken again with py 1.4.22 https://bitbucket.org/hpk42/py/diff/py/_process/forkedfunc.py?diff2=37f557844498&at=default

schlamar commented 10 years ago

Yes, I know. We are working something out. For now, you can pin py to 1.4.21.

mocksoul commented 10 years ago

pytest 2.6.0 depends on py 1.4.22, so you require to downgrade them both =)

My 2 years old patch is still working:

In session start record current PID
Before test start compare current pid with session pid and if they are different -- make multiprocessing start
Before teardown check is we used multiprocessing and stop it if needed

But I if you want things to make right you probably should ask xdist to provide box_start/box_stop hooks which you will be able to use.

schlamar commented 10 years ago

@mocksoul Do you have some code? =)

aconrad commented 10 years ago

Do we have a fix for this?

schlamar commented 10 years ago

@aconrad #15 should fix it, I might find some time at the weekend to have a look at it.

schlamar commented 10 years ago

fixed (again) in 1633bd354f17ed9906a110d9f7ced69921b70f47, will be released soon

aconrad commented 10 years ago

Sweet! @schlamar can you ping this ticket when a new release is available? Thanks!

mocksoul commented 10 years ago

As being said, it is fixed in pytest-cov-1.8.0, https://pypi.python.org/pypi/pytest-cov/1.8.0

pytest-dev / pytest-cov

coverage is wrong when running with xdist's --boxed #9