Custom Plugin how to get Subtest Details?

[mkmoisen]

I have a custom plugin that takes the results of my tests and saves them to a database.

When I use pytest, the custom plugin doesn't seem to get the subtest details.

Here is an example of the hooks that I'm using.

def pytest_collection_modifyitems(session, config, items):
    for item in items:
        print(item.nodeid, str(item.path), item.parent.name)

@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item, call):
    outcome = yield

    report = outcome.get_result()

    print(item.nodeid, report.outcome)

Each of these items is for a top level test, not a subtest.

Is it possible to get access to the subtest level details?

[nicoddemus]

If you want to access the report, implement the pytest_runtest_logreport hook instead, this is the mechanism plugins use in general to receive test reports (for example the internal terminal plugin implements this hook in order to print test progress).

[mkmoisen]

@nicoddemus Ok it seems like the hooks like pytest_runtest_logreport will not show SubTestReports when I subclass TestCase and use self.subTest(). It only works when I use pytests style with the subtests fixture.

Here is an example test:

from unittest import TestCase

class TestFoo(TestCase):
    def test_all(self):
        with self.subTest('test_foo'):
            raise Exception('foo')
        with self.subTest('test_bar'):
            raise Exception('bar')

def test_baz(subtests):
    with subtests.test('test_baz_first'):
        raise Exception('first')
    with subtests.test('test_baz_second'):
        raise Exception('second')

And a plugin:

def pytest_runtest_logreport(report):
    print(type(report), report)

Running my test results in:

test_foo.py <class '_pytest.reports.TestReport'> <TestReport 'test_foo.py::TestFoo::test_all' when='setup' outcome='passed'>

.<class '_pytest.reports.TestReport'> <TestReport 'test_foo.py::TestFoo::test_all' when='call' outcome='passed'>

<class '_pytest.reports.TestReport'> <TestReport 'test_foo.py::TestFoo::test_all' when='teardown' outcome='passed'>

<class '_pytest.reports.TestReport'> <TestReport 'test_foo.py::test_baz' when='setup' outcome='passed'>

u<class 'pytest_subtests.plugin.SubTestReport'> SubTestReport(context=SubTestContext(msg='test_baz_first', kwargs={}))

u<class 'pytest_subtests.plugin.SubTestReport'> SubTestReport(context=SubTestContext(msg='test_baz_second', kwargs={}))

.<class '_pytest.reports.TestReport'> <TestReport 'test_foo.py::test_baz' when='call' outcome='passed'>

<class '_pytest.reports.TestReport'> <TestReport 'test_foo.py::test_baz' when='teardown' outcome='passed'>

Note how this will only let me get access to subtests created with the pytest subtests fixtures, but it will not let me get access to subclasses of UnitTest that use self.subTest().

Do you have any suggestions?

I have some old code that is using TestCase which I would prefer not to rewrite in the pytest style if possible.

[nicoddemus]

Not sure, this works for me:

λ pytest .tmp\test_ut_report.py -s
============================================================================================= test session starts ==============================================================================================
platform win32 -- Python 3.12.2, pytest-8.2.2, pluggy-1.5.0
rootdir: e:\projects\pytest-subtests
configfile: pytest.ini
plugins: subtests-0.13.1.dev3+gcbff3e1.d20240713
collected 1 item

.tmp\test_ut_report.py <class '_pytest.reports.TestReport'> .tmp/test_ut_report.py::TestFoo::test_all
u<class 'pytest_subtests.plugin.SubTestReport'> .tmp/test_ut_report.py::TestFoo::test_all
u<class 'pytest_subtests.plugin.SubTestReport'> .tmp/test_ut_report.py::TestFoo::test_all
.<class '_pytest.reports.TestReport'> .tmp/test_ut_report.py::TestFoo::test_all
<class '_pytest.reports.TestReport'> .tmp/test_ut_report.py::TestFoo::test_all

🤔

[mkmoisen]

@nicoddemus

Interestingly, it was not printing because I was not running with -s flag. But the same is not true when I use a pytest style function with the subtests fixture - it prints with or without the -s flag.

Thanks for your help on this.

If it is convenient, it might be a good idea to somehow make this -s behavior consistent with pytest style functions and UnitTest subclasses.

conftest.py

def pytest_runtest_logreport(report):
    print(type(report), report)

test_foo.py

from unittest import TestCase

def pytest_runtest_logreport(report):
    print(type(report), report)
    print(dir(report))
    print('')

class TestFoo(TestCase):
    def test_all(self):
        with self.subTest('test_foo'):
            raise Exception('foo')
        with self.subTest('test_bar'):
            raise Exception('bar')

def test_baz(subtests):
    with subtests.test('test_baz_first'):
        raise Exception('first')
    with subtests.test('test_baz_second'):
        raise Exception('second')

pytest test_foo.py

sh-4.4$ pytest test_foo.py 
===================================================================================== test session starts ======================================================================================
platform linux -- Python 3.12.4, pytest-8.2.1, pluggy-1.5.0
rootdir: /opt/app/src/foo
plugins: cov-5.0.0, html-4.1.1, metadata-3.1.1, rerunfailures-14.0, subtests-0.13.0, timer-1.0.0, xdist-3.5.0
collected 2 items                                                                                                                                                                              

test_foo.py <class '_pytest.reports.TestReport'> <TestReport 'test_foo.py::TestFoo::test_all' when='setup' outcome='passed'>
.<class '_pytest.reports.TestReport'> <TestReport 'test_foo.py::TestFoo::test_all' when='call' outcome='passed'>
<class '_pytest.reports.TestReport'> <TestReport 'test_foo.py::TestFoo::test_all' when='teardown' outcome='passed'>
<class '_pytest.reports.TestReport'> <TestReport 'test_foo.py::test_baz' when='setup' outcome='passed'>
u<class 'pytest_subtests.plugin.SubTestReport'> SubTestReport(context=SubTestContext(msg='test_baz_first', kwargs={}))
u<class 'pytest_subtests.plugin.SubTestReport'> SubTestReport(context=SubTestContext(msg='test_baz_second', kwargs={}))
.<class '_pytest.reports.TestReport'> <TestReport 'test_foo.py::test_baz' when='call' outcome='passed'>
<class '_pytest.reports.TestReport'> <TestReport 'test_foo.py::test_baz' when='teardown' outcome='passed'>

pytest test_foo.py -s

sh-4.4$ pytest test_foo.py -s
===================================================================================== test session starts ======================================================================================
platform linux -- Python 3.12.4, pytest-8.2.1, pluggy-1.5.0
rootdir: /opt/app/src/foo
plugins: cov-5.0.0, html-4.1.1, metadata-3.1.1, rerunfailures-14.0, subtests-0.13.0, timer-1.0.0, xdist-3.5.0
collected 2 items                                                                                                                                                                              

test_foo.py <class '_pytest.reports.TestReport'> <TestReport 'test_foo.py::TestFoo::test_all' when='setup' outcome='passed'>
u<class 'pytest_subtests.plugin.SubTestReport'> SubTestReport(context=SubTestContext(msg='test_foo', kwargs={}))
u<class 'pytest_subtests.plugin.SubTestReport'> SubTestReport(context=SubTestContext(msg='test_bar', kwargs={}))
.<class '_pytest.reports.TestReport'> <TestReport 'test_foo.py::TestFoo::test_all' when='call' outcome='passed'>
<class '_pytest.reports.TestReport'> <TestReport 'test_foo.py::TestFoo::test_all' when='teardown' outcome='passed'>
<class '_pytest.reports.TestReport'> <TestReport 'test_foo.py::test_baz' when='setup' outcome='passed'>
u<class 'pytest_subtests.plugin.SubTestReport'> SubTestReport(context=SubTestContext(msg='test_baz_first', kwargs={}))
u<class 'pytest_subtests.plugin.SubTestReport'> SubTestReport(context=SubTestContext(msg='test_baz_second', kwargs={}))
.<class '_pytest.reports.TestReport'> <TestReport 'test_foo.py::test_baz' when='call' outcome='passed'>
<class '_pytest.reports.TestReport'> <TestReport 'test_foo.py::test_baz' when='teardown' outcome='passed'>

[nicoddemus]

If it is convenient, it might be a good idea to somehow make this -s behavior consistent with pytest style functions and UnitTest subclasses.

Indeed we suspend the output capture before reporting in subtests fixtures:

https://github.com/pytest-dev/pytest-subtests/blob/cbff3e1547238584a2cf9be9aa0b0da8d8cb9bc3/src/pytest_subtests/plugin.py#L270-L271

But do not suspend for self.subTest():

https://github.com/pytest-dev/pytest-subtests/blob/cbff3e1547238584a2cf9be9aa0b0da8d8cb9bc3/src/pytest_subtests/plugin.py#L117-L119

The former was implemented in https://github.com/pytest-dev/pytest-subtests/pull/10, might have been just an oversight that the same handling was not done for self.subTest.

Would you be so kind as to open a separate issue with the difference in handling -s between the fixture and TestCase? Thanks!

[mkmoisen]

@nicoddemus

Done, please see here

[nicoddemus]

Thanks @mkmoisen appreciate it