Support unittest.subTest context manager (Python 3.4+)

[dougthor42]

Summary

Python 3.4 added the unittest.subTest() context manager. It would be wonderful if Green elevated the information from subTest() up to the top level.

I think it's best displayed with an example.

Example

Code (taken directly from the subTest() docs):

class NumbersTest(unittest.TestCase):

    def test_even(self):
        """
        Test that numbers between 0 and 5 are all even.
        """
        for i in range(0, 6):
            with self.subTest(i=i):
                self.assertEqual(i % 2, 0)

Green's output:

Unittest's output:

See how the standard unittest module will show you (i=1), (i=3), and (i=5) on the fail line?

Mockup:

Here's a mockup of how Green might look when utilizing the subTest feature:

green project -vvv

I don't think that the subTest items need to be displayed for lower verbosity levels. I also don't think that each failed iteration needs the Traceback (like is done with unittest) unless verbosity is set to the highest level.

[CleanCut]

This would be a great feature! Thanks for bringing it up. I had no idea such a thing existed. If you have an implementation, I'd be happy to review or help with it.

If not, I'll take a stab at implementing this the next time I'm going through a feature-implementation cycle.

[mairsbw]

I'm interested in this. I've written substantial tests and since our codebase is all Python3.5+, I took advantage of this feature before I realized that Green didn't support it very well.

Anything I can do to move this issue along? I might be able to spare some cycles to help write some LoC to get this implemented, but it'd be helpful if someone could point me somewhere in the codebase to start, maybe with some tests?

[CleanCut]

@mairsbw

Since we call into unittest's code whenever we can already, it may be that we get lucky and only have to alter our output function to check for new information. That would be in green/result.py near the bottom in the GreenTestResult class in the printErrors(...) method. You may have to check Python 3.4+'s unittest module source code to see what information it's getting from where.

[mairsbw]

Alright, so I've started looking into this a bit. I just kinda dove right in, so I don't really know the right approach here, but this is what I've found.

There's a new addSubTest() method on the unittest.TestCase object. I figured this would be called because in the docs it says:

Called when a subtest finishes.

I'm running the test that @dougthor42 suggested (along with two of my own) through this library, but I never see the addSubTest() method being called, only the addFailure() function gets called. This corresponds with the comment in the addSubTest() docs that say that

The default implementation does nothing when the outcome is a success, and records subtest failures as normal failures.

So it's unclear to me if this is a Python bug or just that their docs aren't actually clear because addSubTest() is never called for either all-passing tests (using subTest) or partially-failing subtests.

@CleanCut Could you offer any suggestions at this point? I'm not very familiar with green or the unittest workings and maybe this is enough info for you to point me in the right direction?

[CleanCut]

@mairsbw

You need to read the unittest module source code and find what will cause it to start calling addSubTest(), and then make sure that whatever it needs to happen happens.

That's what I did to get the rest of the unittest-integration stuff working, anyway.

The unittest module is all implemented in python. On OS X and Linux that python source is viewable in plaintext. Maybe it is on Windows too. This is an easy way to find where the module is on disk for your own python installation:

$ python
Python 2.7.10 (default, Oct 23 2015, 19:19:21)
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import unittest
>>> unittest.__file__
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/__init__.pyc'

[VladV]

@mairsbw @CleanCut

Hello. Is there some progress on this issue?

I recently found Green and liked it very much, but lack of this feature makes me stick with (somewhat ugly) native unittest runner. I'd be glad to help with the implementation, just wanted to find out if anything is done already.

Thanks.

[CleanCut]

@VladV I haven't heard of any work on this particular feature since @mairsbw's last comment. You are certainly welcome to give implementing it a try!

[VladV]

I've implemented a preliminary version, please have a look. Output is like this:

$ python.exe -m green.cmdline -vv
test_green
  T
F   test_1 (i=1, j=1)
F   test_1 (i=3, j=3)

Failure in test_green.T.test_1 (i=1, j=1)
  File "C:\Python36\lib\unittest\case.py", line 59, in testPartExecutor
    yield
  File "C:\Python36\lib\unittest\case.py", line 519, in subTest
    yield
  File "C:\Users\vladvolkov\PycharmProjects\Test\test_green.py", line 8, in test_1
    self.assertEqual(0, i % 2)
  File "C:\Python36\lib\unittest\case.py", line 821, in assertEqual
    assertion_func(first, second, msg=msg)
  File "C:\Python36\lib\unittest\case.py", line 814, in _baseAssertEqual
    raise self.failureException(msg)
AssertionError: 0 != 1

Failure in test_green.T.test_1 (i=3, j=3)
  File "C:\Python36\lib\unittest\case.py", line 59, in testPartExecutor
    yield
  File "C:\Python36\lib\unittest\case.py", line 519, in subTest
    yield
  File "C:\Users\vladvolkov\PycharmProjects\Test\test_green.py", line 8, in test_1
    self.assertEqual(0, i % 2)
  File "C:\Python36\lib\unittest\case.py", line 821, in assertEqual
    assertion_func(first, second, msg=msg)
  File "C:\Python36\lib\unittest\case.py", line 814, in _baseAssertEqual
    raise self.failureException(msg)
AssertionError: 0 != 1

Ran 1 test in 0.479s

FAILED (failures=2)

[CleanCut]

Sorry for the big delay, I started a new job two weeks ago and I am in the middle of 4 weeks of training. I'll take a look as soon as I can!

[VladV]

Congratulations on the new job! I'm also struggling to find some time to complete the feature and to test it properly. It seems, there're some problems with capturing stdout/stderr in sub-tests - I'm not sure yet whether they are caused by my changes or not, but I'd like to fix them anyway.

[CleanCut]

Resolved in 2.11.0 (just released).

[lsh-0]

thanks guys, works a treat