Test failure with pytest-3.1.0+: TestLooponFailing.test_looponfail_removed_test

[swt2c]

I'm seeing a test failure which unfortunately only seems to happen when I run on the Fedora build machines (it does not happen when I build locally). Has anyone seen this before? If not, I'll keep digging into it.

============================= test session starts ==============================
platform linux2 -- Python 2.7.13, pytest-3.1.1, py-1.4.34, pluggy-0.4.0
rootdir: /builddir/build/BUILD/pytest-xdist-1.17.1, inifile: tox.ini
plugins: xdist-1.17.1
collected 110 items
testing/acceptance_test.py ..............s.x...xx.............x......
testing/test_boxed.py ..xx..
testing/test_dsession.py ........x..x
testing/test_looponfail.py ...........F.ss
testing/test_newhooks.py ..
testing/test_plugin.py ........
testing/test_remote.py x.....s..x.
testing/test_slavemanage.py ........x....s
=========================== short test summary info ============================
SKIP [1] testing/test_remote.py:185: skip at module level illegal in pytest 3.0
SKIP [3] /builddir/build/BUILDROOT/pytest-3.1.1-1.fc27.noarch/usr/lib/python2.7/site-packages/_pytest/pytester.py:1023: could not import 'pexpect'
SKIP [1] /usr/lib/python2.7/site-packages/_pytest/config.py:1212: no 'gspecs' option found
FAIL testing/test_looponfail.py::TestLooponFailing::()::test_looponfail_removed_test
XFAIL testing/acceptance_test.py::TestDistEach::()::test_simple_diffoutput
  reason: [NOTRUN] other python versions might not have py.test installed
XFAIL testing/acceptance_test.py::test_terminate_on_hangingnode
XFAIL testing/acceptance_test.py::test_session_hooks
  reason: [NOTRUN] works if run outside test suite
XFAIL testing/acceptance_test.py::TestNodeFailure::()::test_each_multiple
  #20: xdist race condition on node restart
XFAIL testing/test_boxed.py::test_functional_boxed_capturing[sys]
  capture cleanup needed
XFAIL testing/test_boxed.py::test_functional_boxed_capturing[fd]
  capture cleanup needed
XFAIL testing/test_dsession.py::TestDistReporter::()::test_rsync_printing
XFAIL testing/test_dsession.py::test_pytest_issue419
  duplicate test ids not supported yet
XFAIL testing/test_remote.py::test_remoteinitconfig
  #59
XFAIL testing/test_remote.py::TestSlaveInteractor::()::test_happy_run_events_converted
  reason: implement a simple test for event production
XFAIL testing/test_slavemanage.py::TestNodeManager::()::test_rsync_roots_no_roots
  reason: [NOTRUN] 
=================================== FAILURES ===================================
________________ TestLooponFailing.test_looponfail_removed_test ________________
self = <test_looponfail.TestLooponFailing instance at 0x7fc4b2ae7098>
testdir = <Testdir local('/tmp/pytest-of-mockbuild/pytest-0/testdir/test_looponfail_removed_test0')>
    def test_looponfail_removed_test(self, testdir):
        modcol = testdir.getmodulecol("""
                def test_one():
                    assert 0
                def test_two():
                    assert 0
            """)
        remotecontrol = RemoteControl(modcol.config)
        remotecontrol.loop_once()
        assert len(remotecontrol.failures) == 2

        modcol.fspath.write(py.code.Source("""
                def test_xxx(): # renamed test
                    assert 0
                def test_two():
                    assert 1 # pass now
            """))
        removepyc(modcol.fspath)
        remotecontrol.loop_once()
>       assert len(remotecontrol.failures) == 0
E       AssertionError: assert 1 == 0
E        +  where 1 = len(['test_looponfail_removed_test.py::test_xxx'])
E        +    where ['test_looponfail_removed_test.py::test_xxx'] = <xdist.looponfail.RemoteControl object at 0x7fc4b2c39e50>.failures
/builddir/build/BUILD/pytest-xdist-1.17.1/testing/test_looponfail.py:215: AssertionError
----------------------------- Captured stdout call -----------------------------
============================= test session starts ==============================
platform linux2 -- Python 2.7.13, pytest-3.1.1, py-1.4.34, pluggy-0.4.0
rootdir: /tmp/pytest-of-mockbuild/pytest-0/testdir/test_looponfail_removed_test0, inifile:
plugins: xdist-1.17.1
collected 0 items
=============================== warnings summary ===============================
None
  Module already imported so can not be re-written: xdist
  Module already imported so can not be re-written: xdist
  Module already imported so can not be re-written: xdist
-- Docs: http://doc.pytest.org/en/latest/warnings.html
========================== 3 warnings in 0.00 seconds ==========================
============================= test session starts ==============================
platform linux2 -- Python 2.7.13, pytest-3.1.1, py-1.4.34, pluggy-0.4.0
rootdir: /tmp/pytest-of-mockbuild/pytest-0/testdir/test_looponfail_removed_test0, inifile:
plugins: xdist-1.17.1
collected 2 items
collected 2 items
test_looponfail_removed_test.py FF
=================================== FAILURES ===================================
___________________________________ test_one ___________________________________
    def test_one():
>       assert 0
E       assert 0
test_looponfail_removed_test.py:2: AssertionError
___________________________________ test_two ___________________________________
    def test_two():
>       assert 0
E       assert 0
test_looponfail_removed_test.py:4: AssertionError
=========================== 2 failed in 0.02 seconds ===========================
============================= test session starts ==============================
platform linux2 -- Python 2.7.13, pytest-3.1.1, py-1.4.34, pluggy-0.4.0
rootdir: /tmp/pytest-of-mockbuild/pytest-0/testdir/test_looponfail_removed_test0, inifile:
plugins: xdist-1.17.1
collected 7 items
collected 7 items
test_looponfail_removed_test.py F.
=================================== FAILURES ===================================
___________________________________ test_xxx ___________________________________
    def test_xxx(): # renamed test
>       assert 0
E       assert 0
test_looponfail_removed_test.py:3: AssertionError
====================== 1 failed, 1 passed in 0.02 seconds ======================
========= 1 failed, 93 passed, 5 skipped, 11 xfailed in 35.09 seconds ==========

[swt2c]

Okay, I figured out why I was only seeing this on the Fedora builders: the builders had been upgraded to pytest-3.1.1, but pytest-3.0.7 was the latest version pushed out to the mirrors. Now that pytest-3.1.1 has been pushed to the mirrors, I can reproduce this error locally. I did a bit more investigation and tried pytest-3.1.0 - the same failure occurs there as well.

Thus, it seems that this failure started with pytest-3.1.0. I am not familiar enough with the code though to figure out why it fails, though.

[timyhou]

I inadvertently experienced this as well while working on a separate issue. If I remember correctly there were some recent changes in pytest with regard to "setup" and gathering tests.

[FRidh]

We've been wanting to upgrade to pytest 3.1.3 but encountered this bug as well. What is the recommendation? Can these failing tests be ignored? Should we stick with pytest 3.0.x?

[nicoddemus]

They are isolated to "loop on failing" AFAICT, so unless you use that feature you should not have problems related to it. I have been using 3.1 at work and the latest xdist at work without issues.