CleanCut / green

Green is a clean, colorful, fast python test runner.
MIT License
785 stars 75 forks source link

`stdout` consumed inside failing subtests #238

Closed jakeanq closed 3 years ago

jakeanq commented 3 years ago

When printing something inside a test that contains a subtest that fails, no output is captured.

Expected behaviour: stdout is captured and printed somewhere relevant to the test, perhaps with the results for all subtests lumped together in a block for the test.

Current behaviour: stdout is printed correctly if all subtests pass, but not if one fails no output is shown. An example to reproduce this is shown below.

test_test.py:

import unittest

class SubTest(unittest.TestCase):
    def test_subtest(self):
        print("Passing subtest: before")
        with self.subTest("A sub-testing"):
            print("Passing subtest: within")
            self.assertTrue(True)

        print("Passing subtest: after")

    def test_failing_subtest(self):
        print("Failing subtest: before")
        with self.subTest("A sub-testing"):
            print("Failing subtest: within")
            self.assertTrue(False)

        print("Failing subtest: after")

    def test_no_subtest(self):
        print("No subtest: before")

        print("No subtest: within")
        self.assertTrue(False)

        print("No subtest: after")

which produces, when run with green (all options left as default):

FF.

Captured stdout for test_test.SubTest.test_subtest
Passing subtest: before
Passing subtest: within
Passing subtest: after

Failure in test_test.SubTest.test_failing_subtest [A sub-testing]
  File "/usr/lib/python3.8/unittest/case.py", line 60, in testPartExecutor
    yield
  File "/usr/lib/python3.8/unittest/case.py", line 582, in subTest
    yield
  File "/home/jake/green_test/test_test.py", line 16, in test_failing_subtest
    self.assertTrue(False)
  File "/usr/lib/python3.8/unittest/case.py", line 765, in assertTrue
    raise self.failureException(msg)
AssertionError: False is not true

Failure in test_test.SubTest.test_no_subtest
  File "/usr/lib/python3.8/unittest/case.py", line 60, in testPartExecutor
    yield
  File "/usr/lib/python3.8/unittest/case.py", line 676, in run
    self._callTestMethod(testMethod)
  File "/usr/lib/python3.8/unittest/case.py", line 633, in _callTestMethod
    method()
  File "/home/jake/green_test/test_test.py", line 24, in test_no_subtest
    self.assertTrue(False)
  File "/usr/lib/python3.8/unittest/case.py", line 765, in assertTrue
    raise self.failureException(msg)
AssertionError: False is not true

Captured stdout for test_test.SubTest.test_no_subtest
No subtest: before
No subtest: within

Ran 3 tests in 0.030s using 8 processes

FAILED (failures=2, passes=1)

green --version: Green 3.2.0, Coverage 5.1, Python 3.8.6

Possible reference to this issue: https://github.com/CleanCut/green/issues/111#issuecomment-321700463

CleanCut commented 3 years ago

SubTests are endlessly annoying. 😝

Thanks for the very nice test-file. That made it straightforward to identify and fix. I just release 3.2.5 with the fix. 👍