Running tests in parallel (like make -j2)

[GoogleCodeExporter]

It would be great if nose had an option to run tests in parallel, similar to 
running 'make -j2'.

To avoid the GIL, presumably you would need to fork off N test runner 
processes, while the main 
process idled to collect test results.  Is this the sort of thing that could be 
implemented as a plugin, 
or would it need changes to the core?

Original issue reported on code.google.com by vols...@gmail.com on 11 Sep 2007 at 2:34

[GoogleCodeExporter]

It should be *mostly* possible with a plugin, since you could either use
prepareTest() to replace the top-level test callable with a function that forks 
and
collects, or (maybe easier) use prepareTestCase() to replace each individual 
test
case with a fork/collect  -- or more realistically with something to place the 
test
case into some kind of shared queue for one of N pre-forked runners to pick up. 

Why only mostly possible: the difficulty you'll run into is handling package or
module-level fixtures when running tests in parallel. I don't see an easy 
solution
for that.

But I think this is a really good idea nonetheless, so I'll keep it on the 
books for
a release after 0.10. Anyone who wants to give it a try, grab this ticket.

Original comment by jpelle...@gmail.com on 11 Sep 2007 at 2:46

• Added labels: Milestone-Release1.0, Type-Enhancement
• Removed labels: Type-Defect

[GoogleCodeExporter]

There was some discussion building a plugin to run tests in parallel, across 
multiple
OS processes and remote machines:
http://groups.google.com/group/nose-users/browse_thread/thread/bc0edcf6a3fc167b/

In summary:

- there should be enough hooks in the nose 0.10 plugin API for this
- making use of 3rd party "parallel" processing modules?
  - the parallel python / pp module (http://www.parallelpython.com/) is probably
*not* a good fit
  - the processing module (http://cheeseshop.python.org/pypi/processing) is a possibility
  - Pyro (http://pyro.sourceforge.net/) is also a possibility
  - testoob (http://testoob.sourceforge.net/) supports testing in multiple threads
using twisted, so this code may provide some useful insight
  - twisted ;) (http://twistedmatrix.com/trac/)

In addition to Jason's comment above, I think another hurdle will be how a 
"worker"
process will communicate test results (what was run, what passed, what failed,
traceback, etc) back to the "supervisor."  Jason posted this link on another 
topic:
http://mail.python.org/pipermail/python-3000/2007-April/006604.html which may 
have
some usable code for generating picklable tracebacks.

Original comment by kumar.mcmillan on 3 Dec 2007 at 7:25

[GoogleCodeExporter]

IPython should also be considered as 3rd party option:
http://ipython.scipy.org/moin/Parallel_Computing

Original comment by kumar.mcmillan on 4 Dec 2007 at 12:00

[GoogleCodeExporter]

There is also a thread now on Testing In Python, kicking around the idea of 
creating
a Pycon 2008 sprint for this project: 
http://lists.idyll.org/pipermail/testing-in-python/2007-December/thread.html#463

Original comment by kumar.mcmillan on 7 Dec 2007 at 7:08

[GoogleCodeExporter]

Here are some insightful updates that were made to the Testing In Python 
thread. 
Both py.test and twisted's trial test runner have experimental features for 
running
distributed tests in parallel :

- py.test: https://codespeak.net/py/dist/test.html#automated-distributed-testing
- trial: http://twistedmatrix.com/trac/ticket/1784

Original comment by kumar.mcmillan on 10 Dec 2007 at 10:43

[GoogleCodeExporter]

Here is a publisher/subscriber implementation (currently in beta) that may 
provide an
out-of-the-box way to publish test results from remote test runners back to a 
central
"test result reporter" :

http://pypubsub.sourceforge.net/

Original comment by kumar.mcmillan on 20 Dec 2007 at 7:16

[GoogleCodeExporter]

will need to create pickle-able tracebacks to send from a worker back to the
supervisor.  This seems to be easy enough, swiping from John J. Lee's code in 
the
nosepipe plugin, which was based on an example from Collin Winter:
http://mail.python.org/pipermail/python-3000/2007-April/006604.html

    def _exc_info_to_string(self, err, test):
        exctype, value, tb = err
        # Skip test runner traceback levels
        while tb and self._is_relevant_tb_level(tb):
            tb = tb.tb_next
        if exctype is test.failureException:
            # Skip assert*() traceback levels
            length = self._count_relevant_tb_levels(tb)
            return ''.join(traceback.format_exception(exctype, value, tb,
                                                      length))
        return ''.join(traceback.format_exception(exctype, value, tb))

    def _is_relevant_tb_level(self, tb):
        return tb.tb_frame.f_globals.has_key('__unittest')

    def _count_relevant_tb_levels(self, tb):
        length = 0
        while tb and not self._is_relevant_tb_level(tb):
            length += 1
            tb = tb.tb_next
        return length

Original comment by kumar.mcmillan on 4 Jan 2008 at 5:27

[GoogleCodeExporter]

I've been thinking more about the setup/teardown problem.  This approach might
address it well enough:

a supervisor loops through a suite and assigns an ID number to each test.  It 
then
divides that by the number of workers and sends each worker a "slice," a range 
of
numbers.  

A more concise example: say that a supervisor cataloged a test suite and has 4
workers and 400 tests.  It would execute 4 runs of the nosetests command, 
something like:

nosetests 1:100
nosetests 101:200
nosetests 201:300
nosetests 301:400

if nose's collector was invoked as usual, then any setup/teardowns encountered 
while
collecting those slices will run.

Original comment by kumar.mcmillan on 4 Jan 2008 at 5:43

[GoogleCodeExporter]

Tentatively scheduled for 0.11.

The slice approach won't work, unfortunately, since we can't know whether a 
given
context fixture is re-entrant. I think it will be necessary instead to divide 
tests
up by minimal fixture context (test, class, module or package) and dispatch the
entire context suite when there are fixtures at that level. I believe this will
require a depth-first collection run, then a 2nd pass to look at the collected 
suites
and see where they can be split.

Original comment by jpelle...@gmail.com on 6 Jun 2008 at 6:24

• Changed state: Accepted
• Added labels: Milestone-Release0.11.0
• Removed labels: Milestone-Release1.0

[GoogleCodeExporter]

ah, right.  However, one approach might be to lazily deploy setup/teardown to 
each
slice, as needed.  That is:

tests/
run setup()...
test_one.py
test_two.py
test_three.py
test_four.py
run teardown()...

then deploy slices like so, deploying fixtures where necessary:

slice 1:

run setup()...
test_one.py
run teardown()...

slice 2:

run setup()...
test_two.py
test_three.py
run teardown()...

slice 3:

run setup()...
test_four.py
run teardown()...

As depth increased, package setup/teardown would have to be stacked and the 
whole
stack deployed to each slice but that could be costly (timewise) if many slices 
were
used.

Original comment by kumar.mcmillan on 6 Jun 2008 at 7:47

[GoogleCodeExporter]

Also it would be great to have simpler functionality - run each test or test 
case in
a separate python interpreter one by one (I know it was discussed earlier -
http://nose.python-hosting.com/ticket/77). I know it would be slow but from my 
point
of view don't care much - now I use shell script to run each test case 
separately and
it's not so slow. Would love to see this built in into nose so there would be 
no need
to create some wrappers. And I think it could be a first step in introducing 
parallel
tests.

Kind regards,
Pawel 

Original comment by 111100...@gmail.com on 13 Oct 2008 at 7:45

[GoogleCodeExporter]

In trunk, closing.

Original comment by jpelle...@gmail.com on 26 Mar 2009 at 8:09

• Changed state: Fixed