subtests error in the subTest decorator.

[come-maiz]

python's unittest now has a cool feature for test scenarios called subTest: https://docs.python.org/3/library/unittest.html#unittest.TestCase.subTest

I'm trying to update some tests from unittest.TestCase to testtools.TestCase, but there are some that use subTest which blocks me from using testtools.

This is the trace from one of the failing tests:

ERROR: test_yaml_valid_epochs (snapcraft.tests.test_yaml.TestYaml)
snapcraft.tests.test_yaml.TestYaml.test_yaml_valid_epochs
----------------------------------------------------------------------
testtools.testresult.real._StringException: Empty attachments:
  pythonlogging:''
  pythonlogging:''-1

Traceback (most recent call last):
  File "/usr/lib/python3.5/unittest/mock.py", line 1157, in patched
    return func(*args, **keywargs)
  File "/home/elopio/workspace/canonical/snapcraft/snapcraft/tests/test_yaml.py", line 669, in test_yaml_valid_epochs
    with self.subTest(key=epoch):
  File "/usr/lib/python3.5/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "/usr/lib/python3/dist-packages/unittest2/case.py", line 535, in subTest
    if not self._outcome.result_supports_subtests:
AttributeError: 'NoneType' object has no attribute 'result_supports_subtests'

I think it would be great if testtools supports subTests.

[rbtcollins]

Indeed, it would be nice, Would love it if you wanted to put a patch forward.

[rbtcollins]

Arguably the bug is in subTest itself - it presumes _outcome exists, but _outcome is a unittest internal attribute that isn't part of the API contract, so we should fix this in CPython's stdlib.

[come-maiz]

I agree that unittest should have a fall back, and that the error sucks because of what they are doing. I'll rename this bug.

[come-maiz]

Oh, noes, it was you who renamed it. I think my name was better :smile:

[nicoddemus]

Filed an upstream issue: https://bugs.python.org/issue34900

[nicoddemus]

https://github.com/python/cpython/pull/9707 👍

[nicoddemus]

That PR has been merged, so I guess this can now be closed. 👍

[marph91]

I'm running in the same error, using Python 3.6.9. The change should be already included in 3.6.8.

Simple reproducer:

testsuite.py

import testtools

class TC(testtools.TestCase):
    def test_subtest(self):
        with self.subTest():
            pass

Output:

> python3 -m unittest testsuite.py 
E
======================================================================
ERROR: test_subtest (testsuite.TC)
testsuite.TC.test_subtest
----------------------------------------------------------------------
testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/workspace/python/test_unittest/testsuite.py", line 32, in test_subtest
    with self.subTest():
  File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/usr/lib/python3.6/site-packages/unittest2/case.py", line 535, in subTest
    if not self._outcome.result_supports_subtests:
AttributeError: 'NoneType' object has no attribute 'result_supports_subtests'

----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (errors=1)

[ecbaldwin]

I'm running into the same thing in 3.8.1.

  File "/code/cascade_auth/tests/unit/db/alembic/test_migrations.py", line 116, in test_each_migration
    with self.subTest("Post upgrade: %s" % test_class.__name__):
  File "/usr/local/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.8/site-packages/unittest2/case.py", line 535, in subTest
    if not self._outcome.result_supports_subtests:
AttributeError: 'NoneType' object has no attribute 'result_supports_subtests'

[FedericoRessi]

The problem here is testools is using unittest2 [1] library that hasn't desired fix implemented.

File "/usr/local/lib/python3.8/site-packages/unittest2/case.py", line 535, in subTest

[1] https://github.com/testing-cabal/testtools/blob/master/requirements.txt

Anyway the above patch is just skipping implementing the feature if self._outcome is None. Therefore an actual fix should make test cases providing some instance for self._outcome attribute to have subtests properly working.

[nicoddemus]

I'm sure a similar patch to https://github.com/python/cpython/pull/9707 can be applied to unittest2 if someone is inclined to open a PR.

[FedericoRessi]

I am proposing to drop using very old unittest2 library for newer python versions. Such library hasn't changed since 2005 :-(