Ichimonji10
commented
9 years ago Why are we dropping DDT in favor of subTest? We're doing so because the former approach generates test data when tests are defined, whereas the latter approach generates test data at run-time.
Need a concrete example? Here's a test that makes use of DDT:
#!/usr/bin/env python
from ddt import ddt, data
from unittest import TestCase
@ddt
class NumbersTest(TestCase):
@data(*range(6))
def test_even(self, num):
self.assertEqual(num % 2, 0)Here's an equivalent example that uses subTest:
#!/usr/bin/env python
from unittest2 import TestCase # or `from unittest` if on Python 3.4+
class NumbersTest(TestCase):
def test_even(self):
for i in range(6):
with self.subTest(i):
self.assertEqual(i % 2, 0)Example output with DDT:
$ python -m unittest test2.py
.F.F.F
======================================================================
FAIL: test_even_2_1 (test2.NumbersTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/ichimonji10/env3/lib/python3.4/site-packages/ddt.py", line 146, in wrapper
return func(self, *args, **kwargs)
File "/home/ichimonji10/tmp/test2.py", line 11, in test_even
self.assertEqual(num % 2, 0)
AssertionError: 1 != 0
…
----------------------------------------------------------------------
Ran 6 tests in 0.001s
FAILED (failures=3)Example output with subTest:
$ python -m unittest2 test.py
======================================================================
FAIL: test_even (test.NumbersTest) [1]
----------------------------------------------------------------------
Traceback (most recent call last):
File "test.py", line 9, in test_even
self.assertEqual(i % 2, 0)
AssertionError: 1 != 0
…
----------------------------------------------------------------------
Ran 1 test in 0.001s
FAILED (failures=3)Regardless of whether the tests are run with DDT or subTest, three failures are reported. This is nice for reporting and tracking purposes. It indicates that test results should integrate well with our existing Jenkins set-up either way. That said:
- When using
subTest, only one test actually executed. Curiously, this means that it's possible to have more test failures than tests. - When using
subTest, test names are always consistent, even when performing testing with randomized inputs. This is great for tracking purposes, and it makes it far easier to parse the test suite with tools like Sphinx.
It's pretty easy to switch from unittest to unittest2. All you need to do is list unittest2 as a dependency in setup.py and do some replacements:
for file in $(grep -RI --files-with-matches unittest .); do
sed -i 's/unittest/unittest2/' "$file";
doneUnfortunately, that's only half the story, and the second half of the story is problematic. Once we've switched to unittest2, we can start dropping DDT and making use of subTest instead. The problem here is that none of the well-known test runners deals nicely with subTest. nosetests, nose2 and
py.test all fail an entire test as soon as a single assertion inside of that test fails, even if the assertion is wrapped in a subTest context manager. That completely destroys the point of using subTest!
We can try using the vanilla test runner that ships with unittest2, like so:
python -m unittest2 …But unittest2 doesn't have any native ability to generate JUnit style XML output. It's possible to address this issue by using unittest2 in combination with a separate test runner. In particular, you can use the unittest-xml-reporting test runner. Unfortunately:
- It doesn't appear possible to call the unittest-xml-reporting test runner (or any test runner at all!) from the command line. Instead, unittest2 can only be configured to use a non-standard test runner from within a Python module. This is doable, but will require some work.
- There is an issue (xmlrunner/unittest-xml-reporting#85) that describes how unittest-xml-reporting doesn't play well with
subTest.
That's a blocker.
My goals here are to:
- Generate test data at run-time. (DDT does not do this.)
- Generate JUnit style XML logs from our test runs.
And I'm not sure how to do this right now.
elyezer
sthirugn
lpramuk
@ddtdecorator current usage:Snip! See comment below.