search

nose-devs / nose

nose is nicer testing for python
http://readthedocs.org/docs/nose/en/latest/
1.36k stars 396 forks source link

Multiprocess KeyErrors that result in broken pipes #997

Open mjmj opened 8 years ago

mjmj commented 8 years ago

We run nose with python selenium and occasionally hit a nose issue with KeyErrors that then turn into broken pipes and we can't recover from.

Python 2.6.6 nosetests --goober --processes=1 --process-timeout=300 --process-restartworker --with-xunitmp Debian 6.0.10

Is this a nose bug? Memory corruption? Something we should be doing to narrow this down?

/usr/local/lib/python2.6/dist-packages/pysel/webdriver.py:146:
.........................................................................................................F......S....................................................................................................Traceback (most recent call last):
  File "/usr/local/bin/nosetests", line 9, in <module>
    load_entry_point('nose==1.3.7', 'console_scripts', 'nosetests')()
  File "/usr/local/lib/python2.6/dist-packages/nose/core.py", line 121, in __init__
    **extra_args)
  File "/usr/lib/python2.6/unittest.py", line 817, in __init__
    self.runTests()
  File "/usr/local/lib/python2.6/dist-packages/nose/core.py", line 207, in runTests
    result = self.testRunner.run(self.test)
  File "/usr/local/lib/python2.6/dist-packages/nose/plugins/multiprocess.py", line 474, in run
    if w.is_alive() and len(w.currentaddr.value) > 0:
  File "/usr/lib/python2.6/multiprocessing/managers.py", line 1005, in get
    return self._callmethod('get')
  File "/usr/lib/python2.6/multiprocessing/managers.py", line 740, in _callmethod
    raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError: 
---------------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.6/multiprocessing/managers.py", line 216, in serve_client
    obj, exposed, gettypeid = id_to_obj[ident]
KeyError: '1a033d0'
---------------------------------------------------------------------------
Process Process-252:
Traceback (most recent call last):
  File "/usr/lib/python2.6/multiprocessing/process.py", line 232, in _bootstrap
    self.run()
  File "/usr/lib/python2.6/multiprocessing/process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python2.6/dist-packages/nose/plugins/multiprocess.py", line 652, in runner
    keyboardCaught, shouldStop, loaderClass, resultClass, config)
  File "/usr/local/lib/python2.6/dist-packages/nose/plugins/multiprocess.py", line 745, in __runner
    currentaddr.value = bytes_('')
  File "/usr/lib/python2.6/multiprocessing/managers.py", line 1007, in set
    return self._callmethod('set', (value,))
  File "/usr/lib/python2.6/multiprocessing/managers.py", line 740, in _callmethod
    raise convert_to_error(kind, result)
RemoteError: 
---------------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.6/multiprocessing/managers.py", line 216, in serve_client
    obj, exposed, gettypeid = id_to_obj[ident]
KeyError: '1a001d0'
---------------------------------------------------------------------------
Process Process-251:
Traceback (most recent call last):
  File "/usr/lib/python2.6/multiprocessing/process.py", line 232, in _bootstrap
Process Process-253:
Traceback (most recent call last):
  File "/usr/lib/python2.6/multiprocessing/process.py", line 232, in _bootstrap
    self.run()
  File "/usr/lib/python2.6/multiprocessing/process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python2.6/dist-packages/nose/plugins/multiprocess.py", line 652, in runner
    self.run()
  File "/usr/lib/python2.6/multiprocessing/process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python2.6/dist-packages/nose/plugins/multiprocess.py", line 652, in runner
    keyboardCaught, shouldStop, loaderClass, resultClass, config)
  File "/usr/local/lib/python2.6/dist-packages/nose/plugins/multiprocess.py", line 745, in __runner
    currentaddr.value = bytes_('')
  File "/usr/lib/python2.6/multiprocessing/managers.py", line 1007, in set
    keyboardCaught, shouldStop, loaderClass, resultClass, config)
  File "/usr/local/lib/python2.6/dist-packages/nose/plugins/multiprocess.py", line 745, in __runner
    currentaddr.value = bytes_('')
  File "/usr/lib/python2.6/multiprocessing/managers.py", line 1007, in set
    return self._callmethod('set', (value,))
  File "/usr/lib/python2.6/multiprocessing/managers.py", line 740, in _callmethod
    raise convert_to_error(kind, result)
RemoteError: 
---------------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.6/multiprocessing/managers.py", line 216, in serve_client
    obj, exposed, gettypeid = id_to_obj[ident]
KeyError: '1a70050'
---------------------------------------------------------------------------
    return self._callmethod('set', (value,))
  File "/usr/lib/python2.6/multiprocessing/managers.py", line 740, in _callmethod
    raise convert_to_error(kind, result)
RemoteError: 
---------------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.6/multiprocessing/managers.py", line 216, in serve_client
    obj, exposed, gettypeid = id_to_obj[ident]
KeyError: '1a6aa90'
---------------------------------------------------------------------------
Process Process-254:
Traceback (most recent call last):
  File "/usr/lib/python2.6/multiprocessing/process.py", line 232, in _bootstrap
    self.run()
  File "/usr/lib/python2.6/multiprocessing/process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python2.6/dist-packages/nose/plugins/multiprocess.py", line 652, in runner
    keyboardCaught, shouldStop, loaderClass, resultClass, config)
  File "/usr/local/lib/python2.6/dist-packages/nose/plugins/multiprocess.py", line 745, in __runner
    currentaddr.value = bytes_('')
  File "/usr/lib/python2.6/multiprocessing/managers.py", line 1007, in set
    return self._callmethod('set', (value,))
  File "/usr/lib/python2.6/multiprocessing/managers.py", line 740, in _callmethod
    raise convert_to_error(kind, result)
RemoteError: 
---------------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.6/multiprocessing/managers.py", line 216, in serve_client
    obj, exposed, gettypeid = id_to_obj[ident]
KeyError: '1a033d0'
---------------------------------------------------------------------------
Process Process-255:
Traceback (most recent call last):
  File "/usr/lib/python2.6/multiprocessing/process.py", line 232, in _bootstrap
    self.run()
  File "/usr/lib/python2.6/multiprocessing/process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python2.6/dist-packages/nose/plugins/multiprocess.py", line 652, in runner
    keyboardCaught, shouldStop, loaderClass, resultClass, config)
  File "/usr/local/lib/python2.6/dist-packages/nose/plugins/multiprocess.py", line 700, in __runner
    for test_addr, arg in iter(get, 'STOP'):
  File "/usr/local/lib/python2.6/dist-packages/nose/plugins/multiprocess.py", line 675, in get
    return testQueue.get(timeout=config.multiprocess_timeout)
  File "<string>", line 2, in get
  File "/usr/lib/python2.6/multiprocessing/managers.py", line 722, in _callmethod
    self._connect()
  File "/usr/lib/python2.6/multiprocessing/managers.py", line 709, in _connect
    conn = self._Client(self._token.address, authkey=self._authkey)
  File "/usr/lib/python2.6/multiprocessing/connection.py", line 143, in Client
    c = SocketClient(address)
  File "/usr/lib/python2.6/multiprocessing/connection.py", line 263, in SocketClient
    s.connect(address)
  File "<string>", line 1, in connect
error: [Errno 2] No such file or directory
Process Process-214:
Traceback (most recent call last):
  File "/usr/lib/python2.6/multiprocessing/process.py", line 232, in _bootstrap
    self.run()
  File "/usr/lib/python2.6/multiprocessing/process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python2.6/dist-packages/nose/plugins/multiprocess.py", line 652, in runner
    keyboardCaught, shouldStop, loaderClass, resultClass, config)
  File "/usr/local/lib/python2.6/dist-packages/nose/plugins/multiprocess.py", line 745, in __runner
    currentaddr.value = bytes_('')
  File "/usr/lib/python2.6/multiprocessing/managers.py", line 1007, in set
    return self._callmethod('set', (value,))
  File "/usr/lib/python2.6/multiprocessing/managers.py", line 725, in _callmethod
    conn.send((self._id, methodname, args, kwds))
IOError: [Errno 32] Broken pipe
Process Process-223:
Traceback (most recent call last):
  File "/usr/lib/python2.6/multiprocessing/process.py", line 232, in _bootstrap
    self.run()
  File "/usr/lib/python2.6/multiprocessing/process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python2.6/dist-packages/nose/plugins/multiprocess.py", line 652, in runner
    keyboardCaught, shouldStop, loaderClass, resultClass, config)
  File "/usr/local/lib/python2.6/dist-packages/nose/plugins/multiprocess.py", line 745, in __runner
    currentaddr.value = bytes_('')
  File "/usr/lib/python2.6/multiprocessing/managers.py", line 1007, in set
    return self._callmethod('set', (value,))
  File "/usr/lib/python2.6/multiprocessing/managers.py", line 725, in _callmethod
    conn.send((self._id, methodname, args, kwds))
IOError: [Errno 32] Broken pipe
jonross commented 8 years ago

Seeing similar thing with Python 2.7.6 and Nose 1.3.7. Have you had any luck diagnosing? FWIW I don't see it on a box with Python 2.7.4 and Nose 1.3.0.

nose.plugins.multiprocess: ERROR: Worker 5 error running test or returning results
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/nose-1.3.7-py2.7.egg/nose/plugins/multiprocess.py", line 714, in __runner
    currentstart.value = time.time()
  File "/usr/local/lib/python2.7/multiprocessing/managers.py", line 1041, in set
    return self._callmethod('set', (value,))
  File "/usr/local/lib/python2.7/multiprocessing/managers.py", line 774, in _callmethod
    raise convert_to_error(kind, result)
RemoteError:
---------------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/multiprocessing/managers.py", line 242, in serve_client
    obj, exposed, gettypeid = id_to_obj[ident]
KeyError: '2539190'
---------------------------------------------------------------------------
Failure: RemoteError (
---------------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/multiprocessing/managers.py", line 242, in serve_client
    obj, exposed, gettypeid = id_to_obj[ident]
KeyError: '2539190'
---------------------------------------------------------------------------) ... ERROR
Traceback (most recent call last):
  File "/usr/local/bin/nosetests", line 9, in <module>
    load_entry_point('nose==1.3.7', 'console_scripts', 'nosetests')()
  File "/usr/local/lib/python2.7/site-packages/nose-1.3.7-py2.7.egg/nose/core.py", line 121, in __init__
    **extra_args)
  File "/usr/local/lib/python2.7/unittest/main.py", line 95, in __init__
    self.runTests()
  File "/usr/local/lib/python2.7/site-packages/nose-1.3.7-py2.7.egg/nose/core.py", line 207, in runTests
    result = self.testRunner.run(self.test)
  File "/usr/local/lib/python2.7/site-packages/nose-1.3.7-py2.7.egg/nose/plugins/multiprocess.py", line 474, in run
    if w.is_alive() and len(w.currentaddr.value) > 0:
  File "/usr/local/lib/python2.7/multiprocessing/managers.py", line 1039, in get
    return self._callmethod('get')
  File "/usr/local/lib/python2.7/multiprocessing/managers.py", line 774, in _callmethod
    raise convert_to_error(kind, result)
mjmj commented 8 years ago

No we worked around the problem by changing our tests so the dictionaries weren’t as large and that seemed to help. Would love to know if you find any more information since I’m sure we’ll run into it again.

On August 10, 2016 at 7:17:54 AM, Jonathan Ross (notifications@github.com) wrote:

Seeing similar thing with Python 2.7.6 and Nose 1.3.7. Have you had any luck diagnosing?

nose.plugins.multiprocess: ERROR: Worker 5 error running test or returning results Traceback (most recent call last): File "/usr/local/lib/python2.7/site-packages/nose-1.3.7-py2.7.egg/nose/plugins/multiprocess.py", line 714, in __runner currentstart.value = time.time() File "/usr/local/lib/python2.7/multiprocessing/managers.py", line 1041, in set return self._callmethod('set', (value,)) File "/usr/local/lib/python2.7/multiprocessing/managers.py", line 774, in _callmethod raise convert_to_error(kind, result)

RemoteError:

Traceback (most recent call last): File "/usr/local/lib/python2.7/multiprocessing/managers.py", line 242, in serve_client obj, exposed, gettypeid = id_to_obj[ident]

KeyError: '2539190'

Failure: RemoteError (

Traceback (most recent call last): File "/usr/local/lib/python2.7/multiprocessing/managers.py", line 242, in serve_client obj, exposed, gettypeid = id_to_obj[ident] KeyError: '2539190' ---------------------------------------------------------------------------) ... ERROR Traceback (most recent call last): File "/usr/local/bin/nosetests", line 9, in load_entry_point('nose==1.3.7', 'console_scripts', 'nosetests')() File "/usr/local/lib/python2.7/site-packages/nose-1.3.7-py2.7.egg/nose/core.py", line 121, in init extra_args) File "/usr/local/lib/python2.7/unittest/main.py", line 95, in init** self.runTests() File "/usr/local/lib/python2.7/site-packages/nose-1.3.7-py2.7.egg/nose/core.py", line 207, in runTests result = self.testRunner.run(self.test) File "/usr/local/lib/python2.7/site-packages/nose-1.3.7-py2.7.egg/nose/plugins/multiprocess.py", line 474, in run if w.is_alive() and len(w.currentaddr.value) > 0: File "/usr/local/lib/python2.7/multiprocessing/managers.py", line 1039, in get return self._callmethod('get') File "/usr/local/lib/python2.7/multiprocessing/managers.py", line 774, in _callmethod raise convert_to_error(kind, result)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/nose-devs/nose/issues/997#issuecomment-238880719, or mute the thread https://github.com/notifications/unsubscribe-auth/AC4PLID8Ano8iii3aAFSD4NmtxvdH5cVks5qed2SgaJpZM4IYMZj .

p00j4 commented 7 years ago

@mjmj apologies for hijacking the thread but since i'm new and it would be great if you can point me in right direct on how did you make "---goober" flag working? I ran setup.py but I get no such option: --goober

p00j4 commented 7 years ago

no worries @mjmj , got to from the author dustinkeitel using (pip install -e .)