Captured output not displayed for functions from the outer test

[pganssle]

Normally when a function fails, the captured output from the whole function is printed in the output results, but with subtests, pytest is only outputting the captured logs and output from the specific subtest that failed. This makes some sense, but I think it would make more sense to show the captured output from the parts of the test not under a subtest. For example:

def my_function():
    print("This function is critical!")
    return 3, 4

def test_with_subtests(subtests):
    a, b = my_function()

    with subtests.test(a):
        assert a == 5

    with subtests.test(b):
        assert b == 6

This doesn't print anything, but the logs from my_function may be important, since the results of that function are used in the subtests.

I think it would make sense to display output from anything in the enclosing scope of a given subtest, so for example:

def test_with_subtests(subtests):
    print(1)
    for i in range(5):
        with subtests.test(i):
            print(f"Subtest: {i}")
            assert i < 4

    with subtests.test("Nested"):
        print("Nested")
        with subtests.test("Failing"):
            print("Fail")
            assert False

        with subtests.test("Succeeding"):
           print("Success")
           assert True

I would want this to show

1
4
Nested
Fail

Or if the output is not combined at the test level, then for subtest 4, I'd want:

1
4

And for subest "Nested/Failing" I'd want:

1
Nested
Fail

This is somewhat related to #11 — I think tests should probably be considered failures if any subtests nested under them fail, and this is an example of one reason it makes sense to think of things that way.

[nicoddemus]

Thanks @pganssle for the report.

I think tests should probably be considered failures if any subtests nested under them fail, and this is an example of one reason it makes sense to think of things that way.

Yeah, I agree.

The initial reasoning was to mirror how unittest handles subtests, but I think we can have different output with the subtests fixture, to better match how pytest users expect things to work.

[ncoghlan]

I recently encountered this same issue when using the "break a complex test scenario into multiple subtests" technique described under "Beyond parameterization" in @pganssle's article at https://blog.ganssle.io/articles/2020/04/subtests-in-python.html

(since my subtest assertion steps don't produce any output, I was initially going to post under #11 thinking the output display wasn't triggering at all, but then I saw the cross-reference from here).

I worked around the problem by keeping separate counts of subtests started and subtests passed and then adding a self.assertEqual(subtests_passed, subtests_started, "Failed due to failed subtest(s)") assertion at the end of the parent test case.

That said, I don't think the answer here is as simple as always failing the parent test case when subtests fail, as for the "subtests as an alternative to parameterization" use case, the current behaviour is appropriate.