search

pombreda / python-nose

Automatically exported from code.google.com/p/python-nose
0 stars 0 forks source link

multiprocess functionality does not play way with generators #399

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
What steps will reproduce the problem?

1.create a simple test generator function like:

def test_foo():
    for i in range(10):
        yield foosleep, i

def foosleep(n):
    time.sleep(4)

What is the expected output? What do you see instead?

Using --processes=10, all tests should complete in ~4s.
Instead, it takes 40s since the next test does not start until the last is 
finished.

What version of the product are you using? On what operating system?

1.0.0 on ubuntu 10.04 x86-64

Please provide any additional information below.

i'm guessing we need a way to collect all tests before they are sent to the 
parallelizer?

Original issue reported on code.google.com by ffxvzero@gmail.com on 19 Feb 2011 at 9:04

GoogleCodeExporter commented 9 years ago 
After playing around with the multiprocess.py file a bit, I got a hacked up 
version to add tests from generates back to the queue. This was a bit tricky 
because function args had to be also passed. I'm attaching the new 
multiprocess.py file and its diff, hopefully you guys have a better way of 
implementing this.

Original comment by ffxvzero@gmail.com on 20 Feb 2011 at 4:06

Attachments:

GoogleCodeExporter commented 9 years ago

Original comment by ffxvzero@gmail.com on 20 Feb 2011 at 4:07

Attachments:

GoogleCodeExporter commented 9 years ago 
i forgot to mention that the biggest change that allowed this to happen is 
overriding the ContextSuite.run method with an implementation that queues up 
self._tests to testQueue if len(self._tests) > 1.

rosen diankov,

Original comment by ffxvzero@gmail.com on 20 Feb 2011 at 4:19

GoogleCodeExporter commented 9 years ago 
this fixes several bugs when passing the results back to the main process.

rosen dainkov,

Original comment by ffxvzero@gmail.com on 21 Feb 2011 at 2:59

Attachments:

GoogleCodeExporter commented 9 years ago 
this fixes bugs when the generator function throws an exception

rosen diankov,

Original comment by ffxvzero@gmail.com on 21 Feb 2011 at 4:37

Attachments:

GoogleCodeExporter commented 9 years ago 
It turns out that the timeout feature of the multiprocess plugin was also 
broken. My expectation was that a timeout of 1s would immediately stop all 
tests that exceeded 1s value. But, the current version of multiprocess just put 
time outs on the queues, and just printed "timed out" messages, which is not so 
useful.

I'm attaching yet more patches, that sends SIGINT to each of the workers and 
when a timeout occurs, they gracefully stop their current test and return a 
failure.

Original comment by ffxvzero@gmail.com on 21 Feb 2011 at 9:00

Attachments:

GoogleCodeExporter commented 9 years ago 
one more small change: the time out failures now return a 
multiprocess.TimedOutException exception to be easily identifiable

rosen diankov,

Original comment by ffxvzero@gmail.com on 21 Feb 2011 at 9:17

Attachments:

GoogleCodeExporter commented 9 years ago 
Instead of posting new versions, you can find the latest multiprocess.py with 
these fixes here:

https://openrave.svn.sourceforge.net/svnroot/openrave/trunk/test/noseplugins/mul
tiprocess.py

rosen diankov,

Original comment by ffxvzero@gmail.com on 28 Feb 2011 at 6:21

GoogleCodeExporter commented 9 years ago

Original comment by kumar.mcmillan on 20 Mar 2011 at 6:00

GoogleCodeExporter commented 9 years ago 
Hello.  Thanks for following up with the patches.  One minor comment: please 
avoid decorators like @staticmethod so that the code will compile without 
syntax errors in older pythons (even though this plugin won't be used).  You 
can just double it up like address = staticmethod(address).

Currently with this patch I get this failure in tox -e py26

======================================================================
ERROR: test_multiprocess.test_mp_process_args_pickleable
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/kumar/dev/nose/nose/case.py", line 187, in runTest
    self.test(*self.arg)
  File "/Users/kumar/dev/nose/unit_tests/test_multiprocess.py", line 54, in test_mp_process_args_pickleable
    runner.run(test)
  File "/Users/kumar/dev/nose/nose/plugins/multiprocess.py", line 298, in run
    pickle.dumps(self.config)))
  File "/Users/kumar/dev/nose/unit_tests/test_multiprocess.py", line 23, in __init__
    self.pickled = pickle.dumps(pargs)
  File "/usr/local/Cellar/python2.6/2.6.5/lib/python2.6/pickle.py", line 1366, in dumps
    Pickler(file, protocol).dump(obj)
  File "/usr/local/Cellar/python2.6/2.6.5/lib/python2.6/pickle.py", line 224, in dump
    self.save(obj)
  File "/usr/local/Cellar/python2.6/2.6.5/lib/python2.6/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/Cellar/python2.6/2.6.5/lib/python2.6/pickle.py", line 562, in save_tuple
    save(element)
  File "/usr/local/Cellar/python2.6/2.6.5/lib/python2.6/pickle.py", line 306, in save
    rv = reduce(self.proto)
  File "/usr/local/Cellar/python2.6/2.6.5/lib/python2.6/multiprocessing/sharedctypes.py", line 185, in __reduce__
    assert_spawning(self)
  File "/usr/local/Cellar/python2.6/2.6.5/lib/python2.6/multiprocessing/forking.py", line 25, in assert_spawning
    ' through inheritance' % type(self).__name__
RuntimeError: Synchronized objects should only be shared between processes 
through inheritance

Also, can you add a test case using the generator example above?  If you get 
stuck, I can help with that.

To run the test suite, install tox and type tox -e py26 at the root of the nose 
source.  http://codespeak.net/tox/

Original comment by kumar.mcmillan on 20 Mar 2011 at 6:19

GoogleCodeExporter commented 9 years ago 
this seems like a simple problem to solve, i'll patch it in a day or two and 
test with tox.

about staticmethod, i'll just make them regular methods that don't use self

about decorator test case...... i looked at the test cases for the current 
multiprocess plugin and it will take me some time to figure things out and 
write a correct test case. perhaps you could help?

Original comment by ffxvzero@gmail.com on 20 Mar 2011 at 6:26

GoogleCodeExporter commented 9 years ago 
hi kumar,

it doesn't look like you were using the latest code from:

https://openrave.svn.sourceforge.net/svnroot/openrave/trunk/test/noseplugins/mul
tiprocess.py

Original comment by ffxvzero@gmail.com on 21 Mar 2011 at 4:11

GoogleCodeExporter commented 9 years ago 
Hi Rosen.  I tried out this link but I got the same pickle test failure as 
above.  That's the main thing to fix since I'm not entirely sure why that fails.

As for creating a regression test, I can try to make one so consider that low 
priority if you get stuck on it.

Original comment by kumar.mcmillan on 21 Mar 2011 at 3:12

GoogleCodeExporter commented 9 years ago 
adding empty comment to trick groups into resending previous messages which I 
accidentally moderated into the trash.

Original comment by jpelle...@gmail.com on 21 Mar 2011 at 3:16

GoogleCodeExporter commented 9 years ago 
ok, this took me a while to figure out, but the problem was not in the new 
multiprocess.py plugin, it was in the test_multiprocess.py test. The original 
one was ignoring the first 4 parameters when pickling (these are the sync 
primitives). The new one has increased sync primitives, so it has to ignore the 
first 7 parameters when pickling. It was also not popping stuff off the 
testQueue, causing an infinite loop. I'm attaching the new test_multiprocess.py

You should get the latest multiprocess.py from the url above (doc updates).

Original comment by rosen.di...@gmail.com on 25 Mar 2011 at 5:45

Attachments:

GoogleCodeExporter commented 9 years ago 
Hi Rosen, thanks for digging into the tests.  This patch is almost there.  The 
python 2.6 tests are fixed but when I patched with the latest code from your 
svn link above I got a failure in python 3 that I don't understand.  Can you 
take a look?

I cleaned up your patch for PEP8 and other minor things so please submit a diff 
against my fork here:
https://bitbucket.org/kumar303/nose-multi

Are you able to run tox -e py32?

(also same failure with py31)

http://pastebin.com/hGhJyVjj

    Traceback (most recent call last):
      ...
      File "/Users/kumar/dev/nose/build/tests/nose/plugins/multiprocess.py", line 318, in run
        currentaddr = Array('c',' '*1000)
      File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/multiprocessing/__init__.py", line 256, in Array
        return Array(typecode_or_type, size_or_initializer, **kwds)
      File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/multiprocessing/sharedctypes.py", line 110, in Array
        obj = RawArray(typecode_or_type, size_or_initializer)
      File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/multiprocessing/sharedctypes.py", line 87, in RawArray
        result.__init__(*size_or_initializer)
    TypeError: one character string expected

I don't know why Array('c',' '*1000) works in 2.6 but not 3.  Can you think of 
a workaround?

Original comment by kumar.mcmillan on 27 Mar 2011 at 5:59

GoogleCodeExporter commented 9 years ago 
that's weird, i just tried it on python 3.1 without problems:

{{{
[TOX] py31: commands succeeded
[TOX] congratulations :)
}}}

And Array does work:

{{{
$ python3.1
Python 3.1.2 (r312:79147, Sep 27 2010, 09:57:50) 
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import multiprocessing
>>> multiprocessing.Array('c',' '*1000)
<SynchronizedString wrapper for <multiprocessing.sharedctypes.c_char_Array_1000 
object at 0xd89560>>
}}}

Unfortunately my linux distro does not have a python 3.2 package so i can't 
test it...

Original comment by ffxvzero@gmail.com on 28 Mar 2011 at 4:15

GoogleCodeExporter commented 9 years ago 
oh, it does look like this problem is limited to Python 3.2 actually (my bad).

On Linux, you can just download the Python 3 source and type ./configure && 
make && make install  -- it will not clobber your existing python and you'll 
get a binary as /usr/local/bin/python3.2  You can pass a flag to configure to 
install it in a custom place for temporary use.  For it to work in tox all you 
have to do is make sure python3.2 is on $PATH

Original comment by kumar.mcmillan on 28 Mar 2011 at 4:00

GoogleCodeExporter commented 9 years ago 
Sorry for the late response, I think i found a common ground. here are the 
patches

Original comment by ffxvzero@gmail.com on 1 Apr 2011 at 8:28

Attachments:

GoogleCodeExporter commented 9 years ago 
Hi Rosen, this is still failing and actually I don't see any differences in 
your patch.  I had to apply it by hand though because you didn't diff against 
the branch I'm working in: https://bitbucket.org/kumar303/nose-multi (maybe I 
missed something?)

Can you clone that branch above, run tox -e py32, and take a look?  I get the 
same thing:

======================================================================
ERROR: test_multiprocess.test_mp_process_args_pickleable
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/kumar/dev/nose-multi/build/tests/nose/case.py", line 188, in runTest
    self.test(*self.arg)
  File "/Users/kumar/dev/nose-multi/build/tests/unit_tests/test_multiprocess.py", line 58, in test_mp_process_args_pickleable
    runner.run(test)
  File "/Users/kumar/dev/nose-multi/build/tests/nose/plugins/multiprocess.py", line 318, in run
    currentaddr = Array('c',' '*1000)
  ...
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/multiprocessing/sharedctypes.py", line 87, in RawArray
    result.__init__(*size_or_initializer)
TypeError: one character string expected

Original comment by kumar.mcmillan on 8 Apr 2011 at 10:19

GoogleCodeExporter commented 9 years ago 
i just noticed that the patch is in reverse! ;0)

the correct code is:
{{{
... Array('c',1000)
... value = b''
}}}

Original comment by ffxvzero@gmail.com on 9 Apr 2011 at 1:44

GoogleCodeExporter commented 9 years ago 
You can apply reverse patches by passing the '-R' option to patch

Original comment by ffxvzero@gmail.com on 9 Apr 2011 at 1:47

GoogleCodeExporter commented 9 years ago 
After some tweaks to make multiprocessing use byte values in all versions of 
Python, the tests are finally passing.

Can you try installing Nose from the branch to see if it works in some real 
world multiprocessing suites?

pip install -e hg+https://bitbucket.org/kumar303/nose-multi#egg=nose

Original comment by kumar.mcmillan on 18 Apr 2011 at 4:28

GoogleCodeExporter commented 9 years ago 
If you made the exact changes i gave you, then it shouldn't be a problem. i've 
already tested them.

Using the command you gave gives an error:

{{{
rdiankov@rdiankov-laptop:~/python-nose-test$ pip install -e 
hg+https://bitbucket.org/kumar303/nose-multi#egg=nose
Checking out nose from hg+https://bitbucket.org/kumar303/nose-multi#egg=nose 
checkout from hg+https://bitbucket.org/kumar303/nose-multi#egg=nose
  Cloning hg https://bitbucket.org/kumar303/nose-multi to ./src/nose
  Running setup.py egg_info for package nose
    no previously-included directories found matching 'doc/.build'
Exception:
Traceback (most recent call last):
  File "/usr/lib/python2.6/dist-packages/pip.py", line 252, in main
    self.run(options, args)
  File "/usr/lib/python2.6/dist-packages/pip.py", line 408, in run
    requirement_set.install_files(finder, force_root_egg_info=self.bundle)
  File "/usr/lib/python2.6/dist-packages/pip.py", line 1786, in install_files
    finder.add_dependency_links(req_to_install.dependency_links)
  File "/usr/lib/python2.6/dist-packages/pip.py", line 1426, in dependency_links
    return self.egg_info_lines('dependency_links.txt')
  File "/usr/lib/python2.6/dist-packages/pip.py", line 1405, in egg_info_lines
    data = self.egg_info_data(filename)
  File "/usr/lib/python2.6/dist-packages/pip.py", line 1376, in egg_info_data
    filename = self.egg_info_path(filename)
  File "/usr/lib/python2.6/dist-packages/pip.py", line 1400, in egg_info_path
    assert len(filenames) == 1, "Unexpected files/directories in %s: %s" % (base, ' '.join(filenames))
AssertionError: Unexpected files/directories in 
/home/rdiankov/python-nose-test/src/nose: 
/home/rdiankov/python-nose-test/src/nose/nose.egg-info 
/home/rdiankov/python-nose-test/src/nose/functional_tests/support/ep/Some_plugin
.egg-info

Storing complete log in ./pip-log.txt
}}}

Original comment by ffxvzero@gmail.com on 19 Apr 2011 at 1:22

Attachments:

GoogleCodeExporter commented 9 years ago 
What version of pip do you have?  Mine is 0.8.1 and I do not get that error 
using python 2.6.  Judging from pip.py it looks like you might have a really 
old version.

Original comment by kumar.mcmillan on 19 Apr 2011 at 5:13

GoogleCodeExporter commented 9 years ago 
you were right, it as too old. just updated to pip 1.0 and tested with no 
problems. awesome work!

Original comment by ffxvzero@gmail.com on 19 Apr 2011 at 5:30

GoogleCodeExporter commented 9 years ago 
Excellent, thanks for testing.  This has been merged into nose's main repo and 
will go out in 1.0.1

Original comment by kumar.mcmillan on 19 Apr 2011 at 6:52

GoogleCodeExporter commented 9 years ago 
Hmm, it seems that test_mp_process_args_pickleable() gets caught in an infinite 
loop but not all of the time.  Can you take a look?  try running it in a while 
loop so that you can catch when it gets stuck:

while true; do tox -e py32; done

After KeyboardInterrupt I see the following traceback.  I added some debugging 
and it appears that the task is timing out without sending any results.  I 
upped the timeout but it doesn't fix it.

Note that I am skipping this test for now so that our CI server doesn't explode.

Traceback (most recent call last):
  File "selftest.py", line 59, in <module>
    nose.run_exit()
  File "/Users/kumar/dev/nose/build/tests/nose/core.py", line 118, in __init__
    **extra_args)
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/unittest/main.py", line 124, in __init__
    self.runTests()
  File "/Users/kumar/dev/nose/build/tests/nose/core.py", line 197, in runTests
    result = self.testRunner.run(self.test)
  File "/Users/kumar/dev/nose/build/tests/nose/core.py", line 61, in run
    test(result)
  File "/Users/kumar/dev/nose/build/tests/nose/suite.py", line 177, in __call__
    return self.run(*arg, **kw)
  File "/Users/kumar/dev/nose/build/tests/nose/suite.py", line 224, in run
    test(orig)
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/unittest/suite.py", line 62, in __call__
    return self.run(*args, **kwds)
  File "/Users/kumar/dev/nose/build/tests/nose/suite.py", line 75, in run
    test(result)
  File "/Users/kumar/dev/nose/build/tests/nose/suite.py", line 177, in __call__
    return self.run(*arg, **kw)
  File "/Users/kumar/dev/nose/build/tests/nose/suite.py", line 224, in run
    test(orig)
  File "/Users/kumar/dev/nose/build/tests/nose/suite.py", line 177, in __call__
    return self.run(*arg, **kw)
  File "/Users/kumar/dev/nose/build/tests/nose/suite.py", line 224, in run
    test(orig)
  File "/Users/kumar/dev/nose/build/tests/nose/case.py", line 46, in __call__
    return self.run(*arg, **kwarg)
  File "/Users/kumar/dev/nose/build/tests/nose/case.py", line 134, in run
    self.runTest(result)
  File "/Users/kumar/dev/nose/build/tests/nose/case.py", line 152, in runTest
    test(result)
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/unittest/case.py", line 494, in __call__
    return self.run(*args, **kwds)
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/unittest/case.py", line 442, in run
    self._executeTestPart(testMethod, outcome, isTest=True)
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/unittest/case.py", line 387, in _executeTestPart
    function()
  File "/Users/kumar/dev/nose/build/tests/nose/case.py", line 188, in runTest
    self.test(*self.arg)
  File "/Users/kumar/dev/nose/build/tests/unit_tests/test_multiprocess.py", line 58, in test_mp_process_args_pickleable
    runner.run(test)
  File "/Users/kumar/dev/nose/build/tests/nose/plugins/multiprocess.py", line 346, in run
    timeout=nexttimeout)
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/multiprocessing/queues.py", line 129, in get
    if not self._poll(block and (deadline-time.time()) or 0.0):
KeyboardInterrupt

Original comment by kumar.mcmillan on 19 Apr 2011 at 8:35

GoogleCodeExporter commented 9 years ago 
Just curious, is this only with python 3?

Original comment by ffxvzero@gmail.com on 19 Apr 2011 at 11:44

GoogleCodeExporter commented 9 years ago 
Yeah, it was only happening in 3.1 and 3.2

Original comment by kumar.mcmillan on 20 Apr 2011 at 2:58

GoogleCodeExporter commented 9 years ago 
ok, i can reproduce it on my side. i'm not an expert on python 2 vs 3 
differences, so this might take sometime to find.

Original comment by ffxvzero@gmail.com on 20 Apr 2011 at 3:37

GoogleCodeExporter commented 9 years ago 
btw, after running that while loop in my shell for a good amount of time I 
stopped it and started working on other projects.  Pretty soon Python (all 
versions) started segfaulting!  Oops.  The crash log didn't make much sense, 
like a bad FS/mount pointer or something.  After restarting my system 
everything was fine, just thought I'd point that out.  (This was on Mac OS X 
10.6.7)

Original comment by kumar.mcmillan on 20 Apr 2011 at 5:51

GoogleCodeExporter commented 9 years ago 
interesting......... so that means there are no bugs to fix for the release?

Original comment by ffxvzero@gmail.com on 21 Apr 2011 at 3:24

GoogleCodeExporter commented 9 years ago 
Well, there is definitely a bug somewhere in the wait loop otherwise the pickle 
test would not be failing like this (I'd think).  It would be great if we could 
fix it!  Otherwise, we can release as is and see if other people run into this 
error.  

A quick web search showed that some people already ran into similar situations 
where Nose's multiprocess got stuck so maybe this is a long standing bug.

Original comment by kumar.mcmillan on 21 Apr 2011 at 5:10

GoogleCodeExporter commented 9 years ago 
if you ask me. let's release it now and make it a priority to fix it for 1.0.2. 
there's a lot of cool new features in multiprocess that possibly outweight this 
;0)

Original comment by ffxvzero@gmail.com on 21 Apr 2011 at 5:16

GoogleCodeExporter commented 9 years ago 
Yeah, good point, I agree.  The test that I had to skip was a regression for 
this change, which added pickle support for nose's Config: 
http://code.google.com/p/python-nose/source/detail?r=36f7de8d99a8#  I'd like to 
try and find a better test for that, if not too difficult.  There are some 
other unit tests in TestNoseConfig though.

Original comment by kumar.mcmillan on 21 Apr 2011 at 1:54

GoogleCodeExporter commented 9 years ago 
As long as there's unit coverage for pickling (which is needed for multiprocess 
to work at all on windows) I'm ok with waiting for after 1.0.1 to fix the wait 
loop problem.

Original comment by jpelle...@gmail.com on 21 Apr 2011 at 2:46

GoogleCodeExporter commented 9 years ago 
I'm getting this error on python2.7.1 for the code at "pip install -e 
hg+https://bitbucket.org/kumar303/nose-multi#egg=nose"

Traceback (most recent call last):
  File "/home/bgolemon/venv/nose/bin/nosetests", line 8, in <module>
    load_entry_point('nose==1.0.1.dev', 'console_scripts', 'nosetests')()
  File "/home/bgolemon/venv/nose/src/nose/nose/core.py", line 118, in __init__
    **extra_args)
  File "/tool/pandora64/.package/python-2.7.1/lib/python2.7/unittest/main.py", line 95, in __init__
    self.runTests()
  File "/home/bgolemon/venv/nose/src/nose/nose/core.py", line 197, in runTests
    result = self.testRunner.run(self.test)
  File "/home/bgolemon/venv/nose/src/nose/nose/plugins/multiprocess.py", line 404, in run
    worker_addr = str(w.currentaddr.value,'ascii')
TypeError: str() takes at most 1 argument (2 given)

Original comment by workitha...@gmail.com on 24 Apr 2011 at 3:29

GoogleCodeExporter commented 9 years ago 
On

/home/bgolemon/venv/nose/src/nose/nose/plugins/multiprocess.py", line 404,

Can you replace str(w.currentaddr.value,'ascii')

with

bytes_(w.currentaddr.value,'ascii')

and see if it works?
thanks

Original comment by ffxvzero@gmail.com on 24 Apr 2011 at 3:49

GoogleCodeExporter commented 9 years ago 
Actually i'm not even sure if the ascii is needed... you'll however get a 
b'xxx' for the print.

Original comment by ffxvzero@gmail.com on 24 Apr 2011 at 3:50

GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
This is the minimal test to demonstrate the error:

def test_generator():
    yield check

def check():
    from time import sleep
    sleep(10)

with setup.cfg as:

[nosetests]
processes=1
process-timeout=1

Original comment by workitha...@gmail.com on 24 Apr 2011 at 4:02

GoogleCodeExporter commented 9 years ago 
That makes it better, but there's a spurious KeyboardInterrupt:

> nosetests test_example.py
Process Process-1:
Traceback (most recent call last):
  File "/tool/pandora64/.package/python-2.7.1/lib/python2.7/multiprocessing/process.py", line 232, in _bootstrap
    self.run()
  File "/tool/pandora64/.package/python-2.7.1/lib/python2.7/multiprocessing/process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
  File "/home/bgolemon/venv/nose/src/nose/nose/plugins/multiprocess.py", line 664, in runner
    for test_addr, arg in iter(get, 'STOP'):
  File "/home/bgolemon/venv/nose/src/nose/nose/plugins/multiprocess.py", line 637, in get
    return testQueue.get(timeout=config.multiprocess_timeout)
  File "/tool/pandora64/.package/python-2.7.1/lib/python2.7/multiprocessing/queues.py", line 103, in get
    if not self._poll(block and (deadline-time.time()) or 0.0):
KeyboardInterrupt
E
======================================================================
ERROR: test_example.test_generator
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/bgolemon/venv/nose/src/nose/nose/plugins/multiprocess.py", line 780, in run
    test(orig)
  File "/home/bgolemon/venv/nose/src/nose/nose/case.py", line 45, in __call__
    return self.run(*arg, **kwarg)
  File "/home/bgolemon/venv/nose/src/nose/nose/case.py", line 133, in run
    self.runTest(result)
  File "/home/bgolemon/venv/nose/src/nose/nose/case.py", line 151, in runTest
    test(result)
  File "/tool/pandora64/.package/python-2.7.1/lib/python2.7/unittest/case.py", line 376, in __call__
    return self.run(*args, **kwds)
  File "/tool/pandora64/.package/python-2.7.1/lib/python2.7/unittest/case.py", line 318, in run
    testMethod()
  File "/home/bgolemon/venv/nose/src/nose/nose/case.py", line 187, in runTest
    self.test(*self.arg)
  File "/home/bgolemon/tool/amd/rex/Rev/implementation-eperl/test/test_example.py", line 6, in check
    sleep(10)
TimedOutException: 'test_example.test_generator'

----------------------------------------------------------------------
Ran 1 test in 2.028s

FAILED (errors=1)

Original comment by workitha...@gmail.com on 24 Apr 2011 at 5:24

GoogleCodeExporter commented 9 years ago 
I also get a very strange message when there is a NameError at the top level, 
outside any test. The minimal test to reproduce:

my_undefined_variable

The error message:

Traceback (most recent call last):
  File "/home/bgolemon/venv/nose/bin/nosetests", line 8, in <module>
    load_entry_point('nose==1.0.1.dev', 'console_scripts', 'nosetests')()
  File "/home/bgolemon/venv/nose/src/nose/nose/core.py", line 118, in __init__
    **extra_args)
  File "/tool/pandora64/.package/python-2.7.1/lib/python2.7/unittest/main.py", line 95, in __init__
    self.runTests()
  File "/home/bgolemon/venv/nose/src/nose/nose/core.py", line 197, in runTests
    result = self.testRunner.run(self.test)
  File "/home/bgolemon/venv/nose/src/nose/nose/plugins/multiprocess.py", line 313, in run
    test_addr = self.addtask(testQueue,tasks,case)
  File "/home/bgolemon/venv/nose/src/nose/nose/plugins/multiprocess.py", line 511, in addtask
    test_addr = MultiProcessTestRunner.address(case)
  File "/home/bgolemon/venv/nose/src/nose/nose/plugins/multiprocess.py", line 524, in address
    file, mod, call = test_address(case.context)
  File "/home/bgolemon/venv/nose/src/nose/nose/util.py", line 403, in test_address
    return test.address()
TypeError: unbound method address() must be called with Failure instance as 
first argument (got nothing instead)

Original comment by workitha...@gmail.com on 24 Apr 2011 at 5:30

GoogleCodeExporter commented 9 years ago 
that's normal, it means your test timed out, you'll notice that a 
TimedOutException occured

Original comment by ffxvzero@gmail.com on 24 Apr 2011 at 5:31

GoogleCodeExporter commented 9 years ago 
I was afraid of that.  We've seen this bug but we thought it wasn't serious 
enough to fix for 1.0.1.

By KeyboardInterrupt do you mean that the test got stuck and you had to 
interrupt it?  Or do you mean that the test timed out after a while and these 
are the tracebacks that you got?

Original comment by kumar.mcmillan on 24 Apr 2011 at 5:31

GoogleCodeExporter commented 9 years ago 
As for the NameError, python nose gives an error when executed without 
multiprocess, so i don't think it is a multiprocess plugin specific thing.

Original comment by ffxvzero@gmail.com on 24 Apr 2011 at 5:33

GoogleCodeExporter commented 9 years ago 
Kumar, the KeyboardInterrupt is the way the new multiprocess plugin interrupts 
long-running processes so that no process run over its specified time. The 
output report is normal behavior and i don't see any problems with it.

rosen,

Original comment by ffxvzero@gmail.com on 24 Apr 2011 at 5:36

GoogleCodeExporter commented 9 years ago 
having a timeout of 1s is seriously unrealistic. During this time a new process 
has to be setup, with the python run-time initialized, and stuff has to be read 
off the queue. Obviously this takes more than a second, so a timeout would be 
expected.

If you want these simple examples to finish under 1s, then perhaps we should 
look into optimizing python itself...

Original comment by ffxvzero@gmail.com on 24 Apr 2011 at 5:44

GoogleCodeExporter commented 9 years ago 
@kumar: Those are the tracebacks generated with no interaction. I did not press 
ctrl+c, and so shouldn't see a KeyBoardInterrupt. This is what I mean by 
spurious. A SystemExit exception (which shows no message) would be more 
appropriate.

   As for the NameError ... i don't think it is a multiprocess plugin specific thing.
It certainly is. I didn't think it was necessary to demonstrate, but when the 
multiprocess plugin is not used, we see a much more reasonable error, and nose 
itself doesn't crash. Also the error (in #44) shows that you are calling 
address on the Failure class itself, rather than a Failure instance, which 
can't ever be correct.

Here is the error message in single-process mode. Note that it is an error of 
the test, rather than nose, so we get an 'E' and a nice summary.
> nosetests test_example.py
E
======================================================================
ERROR: Failure: NameError (name 'undefined_variable' is not defined)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/bgolemon/venv/nose/src/nose/nose/loader.py", line 390, in loadTestsFromName
    addr.filename, addr.module)
  File "/home/bgolemon/venv/nose/src/nose/nose/importer.py", line 39, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/home/bgolemon/venv/nose/src/nose/nose/importer.py", line 86, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/home/bgolemon/tool/amd/rex/Rev/implementation-eperl/test/test_example.py", line 4, in <module>
    undefined_variable
NameError: name 'undefined_variable' is not defined

----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (errors=1)

   having a timeout of 1s is seriously unrealistic
minimal test cases are always unrealistic. I assure you that I can reproduce 
the error in a real situation with a 20 second timeout (there was a hung disk 
mount). I don't understand your argument. nose should have good error messages, 
regardless of whether the user gives reasonable input.

Original comment by workitha...@gmail.com on 24 Apr 2011 at 4:49