xfail + xdist + flaky == xdist crash

[digitalresistor]

I have a single test that has a:

@pytest.xfail(reason="Fails on Ubuntu, works on CentOS")

When that is hit in one of the test runners using xdist, the following internalerror is raised:

20:17:40 +0000 INTERNALERROR> Traceback (most recent call last):
20:17:40 +0000 INTERNALERROR>   File "/var/lib/jenkins/slave/workspace/appunit-tests/venv/local/lib/python2.7/site-packages/_pytest/main.py", line 105, in wrap_session
20:17:40 +0000 INTERNALERROR>     session.exitstatus = doit(config, session) or 0
20:17:40 +0000 INTERNALERROR>   File "/var/lib/jenkins/slave/workspace/appunit-tests/venv/local/lib/python2.7/site-packages/_pytest/main.py", line 141, in _main
20:17:40 +0000 INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
20:17:40 +0000 INTERNALERROR>   File "/var/lib/jenkins/slave/workspace/appunit-tests/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
20:17:40 +0000 INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
20:17:40 +0000 INTERNALERROR>   File "/var/lib/jenkins/slave/workspace/appunit-tests/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
20:17:40 +0000 INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
20:17:40 +0000 INTERNALERROR>   File "/var/lib/jenkins/slave/workspace/appunit-tests/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
20:17:40 +0000 INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
20:17:40 +0000 INTERNALERROR>   File "/var/lib/jenkins/slave/workspace/appunit-tests/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 613, in execute
20:17:40 +0000 INTERNALERROR>     return _wrapped_call(hook_impl.function(*args), self.execute)
20:17:40 +0000 INTERNALERROR>   File "/var/lib/jenkins/slave/workspace/appunit-tests/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 250, in _wrapped_call
20:17:40 +0000 INTERNALERROR>     wrap_controller.send(call_outcome)
20:17:40 +0000 INTERNALERROR>   File "/var/lib/jenkins/slave/workspace/appunit-tests/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 280, in get_result
20:17:40 +0000 INTERNALERROR>     _reraise(*ex)  # noqa
20:17:40 +0000 INTERNALERROR>   File "/var/lib/jenkins/slave/workspace/appunit-tests/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 265, in __init__
20:17:40 +0000 INTERNALERROR>     self.result = func()
20:17:40 +0000 INTERNALERROR>   File "/var/lib/jenkins/slave/workspace/appunit-tests/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
20:17:40 +0000 INTERNALERROR>     res = hook_impl.function(*args)
20:17:40 +0000 INTERNALERROR>   File "/var/lib/jenkins/slave/workspace/appunit-tests/venv/local/lib/python2.7/site-packages/xdist/dsession.py", line 539, in pytest_runtestloop
20:17:40 +0000 INTERNALERROR>     self.loop_once()
20:17:40 +0000 INTERNALERROR>   File "/var/lib/jenkins/slave/workspace/appunit-tests/venv/local/lib/python2.7/site-packages/xdist/dsession.py", line 558, in loop_once
20:17:40 +0000 INTERNALERROR>     call(**kwargs)
20:17:40 +0000 INTERNALERROR>   File "/var/lib/jenkins/slave/workspace/appunit-tests/venv/local/lib/python2.7/site-packages/xdist/dsession.py", line 664, in slave_testreport
20:17:40 +0000 INTERNALERROR>     self.sched.mark_test_complete(node, rep.item_index, rep.duration)
20:17:40 +0000 INTERNALERROR>   File "/var/lib/jenkins/slave/workspace/appunit-tests/venv/local/lib/python2.7/site-packages/xdist/dsession.py", line 280, in mark_test_complete
20:17:40 +0000 INTERNALERROR>     self.node2pending[node].remove(item_index)
20:17:40 +0000 INTERNALERROR> ValueError: list.remove(x): x not in list

This kills the entire test suite and no further tests are run. Command used to make all tests flaky:

pytest -v -n 6 --junitxml=junit.xml --cov --cov-report=term-missing --durations=20 --force-flaky --max-runs=3 --no-success-flaky-report