pytest-xdist + pypy hanging on Travis-CI

nicoulaj commented 7 years ago

My tests don't start when using pytest-xdist on Travis CI with pypy/pypy3, crashing and restarting workers for 50 minutes before I cancelled it.

pytest-xdist version: 1.15.0
My job log: https://travis-ci.org/nicoulaj/rainbow/jobs/212673500
My project at failing revision: https://github.com/nicoulaj/rainbow/tree/7a63558ab6f73b2b8f8cc2d8f6d2cca04a5c9c9c
Changing from -n auto to -n 2 seems to work around the issue => related to #9 ?

RonnyPfannschmidt commented 7 years ago

this is indeed related to #9 travis reports wrong cpu counts to containers

nicoddemus commented 7 years ago

I guess this can be close as duplicate then

nicoulaj commented 7 years ago

It does not explain why it would hang with pypy, it still works fine with cPython even with 32 workers...

RonnyPfannschmidt commented 7 years ago

@nicoulaj this may be a pypy bug then, i probably wont find the time to investigate this one

liiight commented 7 years ago

not sure if it's realted to this but our builds have been starting to crash on travis even when -n 2 is set:

$ py.test --cov flexget --cov-report xml -n 3
============================= test session starts ==============================
platform linux2 -- Python 2.7.13, pytest-3.1.3, py-1.4.34, pluggy-0.4.0
rootdir: /home/travis/build/Flexget/Flexget, inifile: setup.cfg
plugins: xdist-1.19.0, forked-0.2, cov-2.2.1, capturelog-0.7
gw0 I / gw1 I / gw2 I
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/_pytest/main.py", line 101, in wrap_session
INTERNALERROR>     config._do_configure()
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/_pytest/config.py", line 912, in _do_configure
INTERNALERROR>     self.hook.pytest_configure.call_historic(kwargs=dict(config=self))
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 750, in call_historic
INTERNALERROR>     self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/pytest_capturelog.py", line 135, in pytest_configure
INTERNALERROR>     config.pluginmanager.register(CaptureLogPlugin(config), '_capturelog')
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/_pytest/config.py", line 258, in register
INTERNALERROR>     ret = super(PytestPluginManager, self).register(plugin, name)
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 370, in register
INTERNALERROR>     self._verify_hook(hook, hookimpl)
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/_pytest/config.py", line 255, in _verify_hook
INTERNALERROR>     self._warn(warning)
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/_pytest/config.py", line 292, in _warn
INTERNALERROR>     self.hook.pytest_logwarning.call_historic(kwargs=kwargs)
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 750, in call_historic
INTERNALERROR>     self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "<remote exec>", line 102, in pytest_logwarning
INTERNALERROR>   File "<remote exec>", line 26, in sendevent
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/execnet/gateway_base.py", line 717, in send
INTERNALERROR>     self.gateway._send(Message.CHANNEL_DATA, self.id, dumps_internal(item))
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/execnet/gateway_base.py", line 1354, in dumps_internal
INTERNALERROR>     return _Serializer().save(obj)
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/execnet/gateway_base.py", line 1372, in save
INTERNALERROR>     self._save(obj)
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/execnet/gateway_base.py", line 1390, in _save
INTERNALERROR>     dispatch(self, obj)
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/execnet/gateway_base.py", line 1475, in save_tuple
INTERNALERROR>     self._save(item)
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/execnet/gateway_base.py", line 1390, in _save
INTERNALERROR>     dispatch(self, obj)
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/execnet/gateway_base.py", line 1471, in save_dict
INTERNALERROR>     self._write_setitem(key, value)
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/execnet/gateway_base.py", line 1465, in _write_setitem
INTERNALERROR>     self._save(value)
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/execnet/gateway_base.py", line 1390, in _save
INTERNALERROR>     dispatch(self, obj)
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/execnet/gateway_base.py", line 1475, in save_tuple
INTERNALERROR>     self._save(item)
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/execnet/gateway_base.py", line 1388, in _save
INTERNALERROR>     raise DumpError("can't serialize %s" % (tp,))
INTERNALERROR> DumpError: can't serialize <class 'py._path.local.LocalPath'>
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/_pytest/main.py", line 101, in wrap_session
INTERNALERROR>     config._do_configure()
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/_pytest/config.py", line 912, in _do_configure
INTERNALERROR>     self.hook.pytest_configure.call_historic(kwargs=dict(config=self))
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 750, in call_historic
INTERNALERROR>     self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/pytest_capturelog.py", line 135, in pytest_configure
INTERNALERROR>     config.pluginmanager.register(CaptureLogPlugin(config), '_capturelog')
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/_pytest/config.py", line 258, in register
INTERNALERROR>     ret = super(PytestPluginManager, self).register(plugin, name)
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 370, in register
INTERNALERROR>     self._verify_hook(hook, hookimpl)
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/_pytest/config.py", line 255, in _verify_hook
INTERNALERROR>     self._warn(warning)
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/_pytest/config.py", line 292, in _warn
INTERNALERROR>     self.hook.pytest_logwarning.call_historic(kwargs=kwargs)
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 750, in call_historic
INTERNALERROR>     self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "<remote exec>", line 102, in pytest_logwarning
INTERNALERROR>   File "<remote exec>", line 26, in sendevent
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/execnet/gateway_base.py", line 717, in send
INTERNALERROR>     self.gateway._send(Message.CHANNEL_DATA, self.id, dumps_internal(item))
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/execnet/gateway_base.py", line 1354, in dumps_internal
INTERNALERROR>     return _Serializer().save(obj)
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/execnet/gateway_base.py", line 1372, in save
INTERNALERROR>     self._save(obj)
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/execnet/gateway_base.py", line 1390, in _save
INTERNALERROR>     dispatch(self, obj)
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/execnet/gateway_base.py", line 1475, in save_tuple
INTERNALERROR>     self._save(item)
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/execnet/gateway_base.py", line 1390, in _save
INTERNALERROR>     dispatch(self, obj)
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/execnet/gateway_base.py", line 1471, in save_dict
INTERNALERROR>     self._write_setitem(key, value)
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/execnet/gateway_base.py", line 1465, in _write_setitem
INTERNALERROR>     self._save(value)
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/execnet/gateway_base.py", line 1390, in _save
INTERNALERROR>     dispatch(self, obj)
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/execnet/gateway_base.py", line 1475, in save_tuple
INTERNALERROR>     self._save(item)
INTERNALERROR>   File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/execnet/gateway_base.py", line 1388, in _save
INTERNALERROR>     raise DumpError("can't serialize %s" % (tp,))
INTERNALERROR> DumpError: can't serialize <class 'py._path.local.LocalPath'>

removing the -n flag completely solves this

nicoddemus commented 7 years ago

Hi @liiight, thanks. Your problem is due to #214 and will be released shortly (#217).

liiight commented 7 years ago

Thanks for the response, looking forward for the fix!

thedrow commented 7 years ago

Should this be closed now that 0.17.1 is released?

nicoddemus commented 7 years ago

I think the original issue "pytest-xdist + pypy hanging on Travis-CI" has not been fixed yet, right?

RonnyPfannschmidt commented 7 years ago

@nicoddemus the original issue seems to relate to the fundamentally broken node restart mechanism as nodes restart things just keep breaking, thus ultimately timeouting the build

@nicoulaj please note that travis has completely broken reporting of available cpus and cant be trusted to provide correct data

nicoddemus commented 7 years ago

the original issue seems to relate to the fundamentally broken node restart mechanism as nodes restart things just keep breaking, thus ultimately timeouting the build

I propose we change the --slave-max-restarts default to some sensible number to avoid this issue.

RonnyPfannschmidt commented 7 years ago

i agree

RonnyPfannschmidt commented 7 years ago

i propose a default of 4 times the node number

nicoddemus commented 7 years ago

Sounds good: pytest-dev/pytest-xdist#226

blueyed commented 5 years ago

Closing as "duplicate" of #226.

pytest-dev / pytest-xdist

pytest-xdist + pypy hanging on Travis-CI #131