Randomize unittest.TestCase test order plugin

[jpellerin]

've written a plugin that randomizes the test order of tests built with the unittest.TestCase framework. I've attached the plugin, a setup file for installation, and a patch against nose Version: 0.11.0.dev-r635. The patch is somewhat relase agnostic as it just adds files, including a test.

The usage is --randomize

The seed is printed at the start so that failures can be replicated in a deterministic way with --randomize --seed=

Comments/corrections welcome

Google Code Info: Issue #: 255 Author: charles....@gmail.com Created On: 2009-04-25T17:16:34.000Z Closed On:

[jpellerin]

0.11 is near release, so the chances of this getting in are very slim. It would be better -- and is generally better -- to release it as a 3rd-party plugin first. Then if it gains a significant following, and it is stable enough to handle nose's long release cycle, it will make sense to pull it into core for a future release.

Google Code Info: Author: jpelle...@gmail.com Created On: 2009-04-26T21:26:39.000Z

[jpellerin]

I installed the randomize plugin on my Debian Linux system (Python 2.5.4) with nose version 0.11.0. The installation was done by downloading randomize.py and setup_randomize.py, then running "python setup_randomize.py install" as the super user. It seemed to install correctly

When I ran my tests from a bash command line using "nosetests --randomize" it printed the seed and ran the expected number of tests.

When I ran my tests from the same command line using --with-xunit, it created nosetests.xml and listed that expected number of tests in the nosetests.xml file.

When I ran my tests from the same command line with --with-xunit --randomize, it printed the seed, printed the expected number of tests run, and created nosetests.xml. However, nosetests.xml listed "0" tests run instead of the expected number of tests run. The nosetests.xml file only contained an entry which said that 0 tests had run, even though the stdout showed the expected number of tests run.

Are there known conflicts between the randomize plugin and the xunit plugin?

Also, I intentionally inserted a failure into one of my tests so I could watch where the "F" character appeared in the output. I was surprised to see that the "F" appeared in the same location each time I ran the tests. Shouldn't the "F" character have moved around due to the randomization of the tests?

I was also surprised to see (in my installation) that --with-randomize did not report the random seed, while --randomize did report the random seed. Is that expected as well?

Google Code Info: Author: mark.ear...@gmail.com Created On: 2009-05-23T01:26:39.000Z

[jpellerin]

I think I may have now grasped my misunderstanding of what is actually being randomized and what is not being randomized.

It appears the randomization happens within a test class, but the order in which test classes are selected is not randomized.

I created two files, test_one.py and test_two.py with tests in each of those files, then verified that nosetests ran test_one then ran test_two. When I ran nosetests --randomize I saw that the "F" which marks my failing test in test_one moved to different places, but it was always on the left side of the output dots.

Have I understood correctly that randomize is varying the order of tests within a class, but not varying the order of test classes? Is there any plan to make it vary the order of test classes as well?

Google Code Info: Author: mark.ear...@gmail.com Created On: 2009-05-23T12:25:04.000Z

[jpellerin]

It would have helped if I'd read the doc comment prominently places at the top of randomize.py. It clearly states "This plugin randomizes the order of tests within a unittest.TestCase class". I should have known that the randomization was only within a TestCase class

Google Code Info: Author: mark.ear...@gmail.com Created On: 2009-05-23T12:33:20.000Z

[jpellerin]

I think randomizing the order of test classes should be just as easy to implement as randomizing the order of tests within a unittest.TestCase class. I'll take a look and see if I can make it more general.

Google Code Info: Author: charles....@gmail.com Created On: 2009-05-23T14:07:41.000Z

[jpellerin]

Since discovering that I have some order-dependent failures in my test suite, I would be very interested in this plugin, provided it randomly orders all tests, and not just within a class. Was any progress made on that front?

Google Code Info: Author: alisonro...@gmail.com Created On: 2010-10-26T17:52:23.000Z

[jonozzz]

How about:

def prepareTestLoader(self, loader):
    loader.sortTestMethodsUsing = lambda x, y: random.choice([-1, 1])

[andresriancho]

I had to modify the original code posted @ Google code in order to work with:

nosetests version 1.1.3

Uploaded the latest version to gist: https://gist.github.com/3844715 , installing should be as easy as downloading the two files to the same location and then running "sudo python setup_randomize.py install".

[andresriancho]

Improvement needed for randomize plugin: Work well with "--with-id". After some tests I found out that when "--randomize" is used, tests are not identified:

a@b:~/workspace/threading2$ nosetests core/data/bloomfilter/  -v --with-id --randomize
Using 296557797 as seed
test_bloom_string (core.data.bloomfilter.tests.test_pybloom.TestBloomFilter) ... ok
test_bloom_int (core.data.bloomfilter.tests.test_pybloom.TestBloomFilter) ... ok
test_bloom_int (core.data.bloomfilter.tests.test_pybloom.TestScalableBloomfilter) ... ok
test_bloom_string (core.data.bloomfilter.tests.test_pybloom.TestScalableBloomfilter) ... ok

vs.

ab:~/workspace/threading2$ nosetests core/data/bloomfilter/  -v --with-id
#171 test_bloom_int (core.data.bloomfilter.tests.test_pybloom.TestBloomFilter) ... ok
#172 test_bloom_string (core.data.bloomfilter.tests.test_pybloom.TestBloomFilter) ... ok
#173 test_bloom_int (core.data.bloomfilter.tests.test_pybloom.TestScalableBloomfilter) ... ok
#174 test_bloom_string (core.data.bloomfilter.tests.test_pybloom.TestScalableBloomfilter) ... ok

Not sure if this is an error in randomize or "id" plugin, but should be fixed before releasing the randomize plugin.