libregrtest: junit-xml output doesn't support subTest

[encukou]

Running the tests with --junit-xml, the result does not include subTest results, as RegressionTestResult in libregrtest.testresult does not implement addSubTest.

There's no standard XML schema for the test results (see testmoapp/junitxml), but I think multiple <error> elements in a <testcase> make sense

[sobolevn]

@encukou do you already work on this? Or can I take this over, please? :)

[encukou]

I just moved it out of my TODO list, thinking it might be fun for someone else :) If you want, please do take over!

[sobolevn]

What happens now for a test like this:

    def test_broken_subtest(self):
        for x in [1, 2, 3]:
            with self.subTest(x=x):
                self.assertEqual(x, 0)

produces:

<testsuites tests="1" errors="0" failures="3"><testsuite start="2024-11-03 09:35:14.984110" tests="1" errors="0" failures="3"><testcase /></testsuite></testsuites>

If we comment # with self.subTest(x=x):, the output changes to:

<testsuites tests="1" errors="0" failures="1"><testsuite start="2024-11-03 09:34:09.426455" tests="1" errors="0" failures="1"><testcase name="test.test_typing.TypeVarTests.test_broken_subtest" status="run" result="completed" time="0.002489"><system-out /><system-err /><failure type="AssertionError" message="AssertionError: 1 != 0&#10;">Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython2/Lib/unittest/case.py", line 58, in testPartExecutor
    yield
  File "/Users/sobolev/Desktop/cpython2/Lib/unittest/case.py", line 660, in run
    self._callTestMethod(testMethod)
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
  File "/Users/sobolev/Desktop/cpython2/Lib/unittest/case.py", line 606, in _callTestMethod
    result = method()
  File "/Users/sobolev/Desktop/cpython2/Lib/test/test_typing.py", line 569, in test_broken_subtest
    self.assertEqual(x, 0)
    ~~~~~~~~~~~~~~~~^^^^^^
  File "/Users/sobolev/Desktop/cpython2/Lib/unittest/case.py", line 916, in assertEqual
    assertion_func(first, second, msg=msg)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sobolev/Desktop/cpython2/Lib/unittest/case.py", line 909, in _baseAssertEqual
    raise self.failureException(msg)
AssertionError: 1 != 0
</failure></testcase></testsuite></testsuites>

Basically, we need to have multiple <failure> nodes inside <testcase> with .subTest calls.

[sobolevn]

I don't quite like that there are multiple <failure> and multiple <system-out> and <system-err> tags:

<testsuites tests="1" errors="0" failures="3"><testsuite start="2024-11-03 14:20:57.276683" tests="1" errors="0" failures="3"><testcase name="test.test_typing.TypeVarTests.test_broken_subtest" status="run" result="completed" time="0.004247"><system-out /><system-err /><failure type="AssertionError" message="AssertionError: 1 != 0&#10;">Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython2/Lib/unittest/case.py", line 58, in testPartExecutor
    yield
  File "/Users/sobolev/Desktop/cpython2/Lib/unittest/case.py", line 556, in subTest
    yield
  File "/Users/sobolev/Desktop/cpython2/Lib/test/test_typing.py", line 570, in test_broken_subtest
    self.assertEqual(x, 0)
    ~~~~~~~~~~~~~~~~^^^^^^
  File "/Users/sobolev/Desktop/cpython2/Lib/unittest/case.py", line 916, in assertEqual
    assertion_func(first, second, msg=msg)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sobolev/Desktop/cpython2/Lib/unittest/case.py", line 909, in _baseAssertEqual
    raise self.failureException(msg)
AssertionError: 1 != 0
</failure><system-out /><system-err /><failure type="AssertionError" message="AssertionError: 2 != 0&#10;">Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython2/Lib/unittest/case.py", line 58, in testPartExecutor
    yield
  File "/Users/sobolev/Desktop/cpython2/Lib/unittest/case.py", line 556, in subTest
    yield
  File "/Users/sobolev/Desktop/cpython2/Lib/test/test_typing.py", line 570, in test_broken_subtest
    self.assertEqual(x, 0)
    ~~~~~~~~~~~~~~~~^^^^^^
  File "/Users/sobolev/Desktop/cpython2/Lib/unittest/case.py", line 916, in assertEqual
    assertion_func(first, second, msg=msg)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sobolev/Desktop/cpython2/Lib/unittest/case.py", line 909, in _baseAssertEqual
    raise self.failureException(msg)
AssertionError: 2 != 0
</failure><system-out /><system-err /><failure type="AssertionError" message="AssertionError: 3 != 0&#10;">Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython2/Lib/unittest/case.py", line 58, in testPartExecutor
    yield
  File "/Users/sobolev/Desktop/cpython2/Lib/unittest/case.py", line 556, in subTest
    yield
  File "/Users/sobolev/Desktop/cpython2/Lib/test/test_typing.py", line 570, in test_broken_subtest
    self.assertEqual(x, 0)
    ~~~~~~~~~~~~~~~~^^^^^^
  File "/Users/sobolev/Desktop/cpython2/Lib/unittest/case.py", line 916, in assertEqual
    assertion_func(first, second, msg=msg)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sobolev/Desktop/cpython2/Lib/unittest/case.py", line 909, in _baseAssertEqual
    raise self.failureException(msg)
AssertionError: 3 != 0
</failure></testcase></testsuite></testsuites>

I think it would be hard to parse. Not sure that it is even supported. See https://github.com/testmoapp/junitxml/blob/1bcb6fb8b620906ec001a2c8e7a7c8e7c563a64b/examples/junit-complete.xml#L99