Py.Test hangs on timeout when running parallel tests with xdist

[kunchamlokesh]

Hi

We are executing around 400 scripts parallel using pytest. In middle of execution at some point one thread will hung & after some time below error will triggered & finally all scripts execution will get hung

What might be the possible reason or is there any different way to trace & solve this issue

~~~~~~~~~~~~~~~~~~~~~~~~~~ Stack of <unknown> (2056) ~~~~~~~~~~~~~~~~~~~~~~~~~~~
  File "C:\Python27\lib\site-packages\execnet\gateway_base.py", line 277, in _perform_spawn
    reply.run()
  File "C:\Python27\lib\site-packages\execnet\gateway_base.py", line 213, in run
    self._result = func(*args, **kwargs)
  File "C:\Python27\lib\site-packages\execnet\gateway_base.py", line 954, in _thread_receiver
    msg = Message.from_io(io)
  File "C:\Python27\lib\site-packages\execnet\gateway_base.py", line 418, in from_io
    header = io.read(9)  # type 1, channel 4, payload 4
  File "C:\Python27\lib\site-packages\execnet\gateway_base.py", line 386, in read
    data = self._read(numbytes-len(buf))

~~~~~~~~~~~~~~~~~~~~~~~~~~ Stack of MainThread (9576) ~~~~~~~~~~~~~~~~~~~~~~~~~~
  File "<string>", line 1, in <module>
  File "<string>", line 8, in <module>
  File "C:\Python27\lib\site-packages\execnet\gateway_base.py", line 1534, in serve
    SlaveGateway(io=io, id=id, _startcount=2).serve()
  File "C:\Python27\lib\site-packages\execnet\gateway_base.py", line 1047, in serve
    self._execpool.integrate_as_primary_thread()
  File "C:\Python27\lib\site-packages\execnet\gateway_base.py", line 259, in integrate_as_primary_thread
    self._perform_spawn(reply)
  File "C:\Python27\lib\site-packages\execnet\gateway_base.py", line 277, in _perform_spawn
    reply.run()
  File "C:\Python27\lib\site-packages\execnet\gateway_base.py", line 213, in run
    self._result = func(*args, **kwargs)
  File "C:\Python27\lib\site-packages\execnet\gateway_base.py", line 1072, in executetask
    do_exec(co, loc)  # noqa
  File "<string>", line 1, in do_exec
  File "<remote exec>", line 202, in <module>
  File "C:\Python27\lib\site-packages\pluggy\__init__.py", line 617, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "C:\Python27\lib\site-packages\pluggy\__init__.py", line 222, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "C:\Python27\lib\site-packages\pluggy\__init__.py", line 216, in <lambda>
    firstresult=hook.spec_opts.get('firstresult'),
  File "C:\Python27\lib\site-packages\pluggy\callers.py", line 180, in _multicall
    res = hook_impl.function(*args)
  File "C:\Python27\lib\site-packages\_pytest\main.py", line 136, in pytest_cmdline_main
    return wrap_session(config, _main)
  File "C:\Python27\lib\site-packages\_pytest\main.py", line 105, in wrap_session
    session.exitstatus = doit(config, session) or 0
  File "C:\Python27\lib\site-packages\_pytest\main.py", line 143, in _main
    config.hook.pytest_runtestloop(session=session)
  File "C:\Python27\lib\site-packages\pluggy\__init__.py", line 617, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "C:\Python27\lib\site-packages\pluggy\__init__.py", line 222, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "C:\Python27\lib\site-packages\pluggy\__init__.py", line 216, in <lambda>
    firstresult=hook.spec_opts.get('firstresult'),
  File "C:\Python27\lib\site-packages\pluggy\callers.py", line 180, in _multicall
    res = hook_impl.function(*args)
  File "<remote exec>", line 65, in pytest_runtestloop
  File "<remote exec>", line 84, in run_one_test
  File "C:\Python27\lib\site-packages\pluggy\__init__.py", line 617, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "C:\Python27\lib\site-packages\pluggy\__init__.py", line 222, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "C:\Python27\lib\site-packages\pluggy\__init__.py", line 216, in <lambda>
    firstresult=hook.spec_opts.get('firstresult'),
  File "C:\Python27\lib\site-packages\pluggy\callers.py", line 180, in _multicall
    res = hook_impl.function(*args)
  File "C:\Python27\lib\site-packages\_pytest\runner.py", line 62, in pytest_runtest_protocol
    runtestprotocol(item, nextitem=nextitem)
  File "C:\Python27\lib\site-packages\_pytest\runner.py", line 76, in runtestprotocol
    reports.append(call_and_report(item, "call", log))
  File "C:\Python27\lib\site-packages\_pytest\runner.py", line 155, in call_and_report
    call = call_runtest_hook(item, when, **kwds)
  File "C:\Python27\lib\site-packages\_pytest\runner.py", line 175, in call_runtest_hook
    return CallInfo(lambda: ihook(item=item, **kwds), when=when)
  File "C:\Python27\lib\site-packages\_pytest\runner.py", line 189, in __init__
    self.result = func()
  File "C:\Python27\lib\site-packages\_pytest\runner.py", line 175, in <lambda>
    return CallInfo(lambda: ihook(item=item, **kwds), when=when)
  File "C:\Python27\lib\site-packages\pluggy\__init__.py", line 617, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "C:\Python27\lib\site-packages\pluggy\__init__.py", line 222, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "C:\Python27\lib\site-packages\pluggy\__init__.py", line 216, in <lambda>
    firstresult=hook.spec_opts.get('firstresult'),
  File "C:\Python27\lib\site-packages\pluggy\callers.py", line 180, in _multicall
    res = hook_impl.function(*args)
  File "C:\Python27\lib\site-packages\_pytest\runner.py", line 106, in pytest_runtest_call
    item.runtest()
  File "C:\Python27\lib\site-packages\_pytest\unittest.py", line 174, in runtest
    self._testcase(result=self)
  File "C:\Python27\lib\unittest\case.py", line 393, in __call__
    return self.run(*args, **kwds)
  File "C:\Python27\lib\unittest\case.py", line 329, in run
    testMethod()
  File "C:\Python27\Lib\site-packages\WebShipping\Utils\selenium_error_report.py", line 59, in decorator
    value = f(*args, **kwargs)
  File "C:\Python27\Lib\site-packages\WebShipping\Tests\RateShopping\RateShop_All_Carriers_List_Services_Weight_Destination_46902_test.py", line 76, in test_fedex_list_expected_services
    self.compare_services(carrier,backup_carrier,dom_services,intl_services,self.wt_lb,'0')
  File "C:\Python27\Lib\site-packages\WebShipping\Tests\RateShopping\RateShop_All_Carriers_List_Services_Weight_Destination_46902_test.py", line 253, in compare_services
    Login.open_web_shipping_settings(self,username(),password())
  File "C:\Python27\Lib\site-packages\WebShipping\LogUtils\logwrap.py", line 71, in decorator
    value = f(*args, **kwargs)
  File "C:\Python27\Lib\site-packages\WebShipping\Shortcuts\Login.py", line 123, in open_web_shipping_settings
    login_safely(unit_test, username, password, returnUrl=url)
  File "C:\Python27\Lib\site-packages\WebShipping\LogUtils\logwrap.py", line 71, in decorator
    value = f(*args, **kwargs)
  File "C:\Python27\Lib\site-packages\WebShipping\Shortcuts\Login.py", line 82, in login_safely
    login(unit_test, username, password)
  File "C:\Python27\Lib\site-packages\WebShipping\LogUtils\logwrap.py", line 71, in decorator
    value = f(*args, **kwargs)
  File "C:\Python27\Lib\site-packages\WebShipping\Shortcuts\Login.py", line 30, in login
    unit_test.wait_page_load(None,None,20)
  File "C:\Python27\Lib\site-packages\WebShipping\Utils\Base_Test.py", line 887, in wait_page_load
    self.wait(wait_seconds).until(EC.page_loaded_completely())
  File "C:\Python27\Lib\site-packages\WebShipping\Utils\wait_extended.py", line 70, in until
    value = method(self._driver)
  File "C:\Python27\Lib\site-packages\WebShipping\Utils\expected_conditions_extended.py", line 94, in __call__
    return driver.execute_script("return document.readyState") == "complete"
  File "C:\Python27\Lib\site-packages\selenium\webdriver\remote\webdriver.py", line 465, in execute_script
    'args': converted_args})['value']
  File "C:\Python27\Lib\site-packages\selenium\webdriver\remote\webdriver.py", line 234, in execute
    response = self.command_executor.execute(driver_command, params)
  File "C:\Python27\Lib\site-packages\selenium\webdriver\remote\remote_connection.py", line 401, in execute
    return self._request(command_info[0], url, body=data)
  File "C:\Python27\Lib\site-packages\selenium\webdriver\remote\remote_connection.py", line 433, in _request
    resp = self._conn.getresponse()
  File "C:\Python27\lib\httplib.py", line 1121, in getresponse
    response.begin()
  File "C:\Python27\lib\httplib.py", line 438, in begin
    version, status, reason = self._read_status()
  File "C:\Python27\lib\httplib.py", line 394, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "C:\Python27\lib\socket.py", line 480, in readline
    data = self._sock.recv(self._rbufsize)

[RonnyPfannschmidt]

without more context i wont be able to provide support debugging this

it looks like a selenium issue however

[kunchamlokesh]

Hi @RonnyPfannschmidt

Thanks for the response.

I too assume that issue is related to selenium beacuse mostly firefox instance are handled by selenium webdriver.

But entire console error triggered is related to execent which is not related to selenium. So to clarify this i have raised this issue

May i know what are the possibilities to trigger 'Stack of ' errors

[nicoddemus]

Hi @kunchamlokesh,

May i know what are the possibilities to trigger 'Stack of ' errors

That's the traceback dump done by pytest-timeout. I assume what's hanging is this part of the traceback:

  File "C:\Python27\lib\socket.py", line 480, in readline
    data = self._sock.recv(self._rbufsize)

As to what could cause this, it is completely application dependent I'm afraid.

[RonnyPfannschmidt]

the hanging readline happens in selenium