Issues counting passing rates on tests which use subTest()

[dvrogozh]

As we discussed in https://github.com/huggingface/transformers/pull/34723#issuecomment-2479470083, we spotted that tests using with self.subTest along with self.skipTest() report confusing passing rates. We need to understand how we should handle this. Below is breakdown of behavior on few syntetic tests.

Note:

• Behavior depends on whether pytest-subtest is installed or not
• Accelerate brings in pytest-subtest into environment, see https://github.com/huggingface/accelerate/blob/c0552c9012a9bae7f125e1df89cf9ee0b0d250fd/setup.py#L25
• Transformers do not depend on pytest-subtest at the moment
• Transformers ci does not have pytest-subtest in the environment at the moment (according to below breakdown this means that if first subtest is skipped other subtests won't be executed)

Synthetic tests:

• ex0.py - subTest() without skipping:

$ cat ex0.py
import unittest
class T(unittest.TestCase):
    def test_foo(self):
        for i in range(7):
            with self.subTest(i=i):
                self.assertLess(i, 3)

• ex1.py - subTest() with skipping:

$ cat ex1.py
import unittest
class T(unittest.TestCase):
    def test_foo(self):
        for i in range(7):
            with self.subTest(i=i):
                if i < 3:
                    self.skipTest(f"bon {i}")
                self.assertLess(i, 3)

		Failed	Passed	Skipped
No pytest-subtests	ex0.py	1	0	0
No pytest-subtests	ex1.py	0	0	1
With pytest-subtests	ex0.py	4	1	0
With pytest-subtests	ex1.py	1	1	3

Logs:

• With pytest-subtests:

$ python -m pytest ex0.py
...
(i=3) SUBFAIL ex0.py::T::test_foo - AssertionError: 3 not less than 3
(i=4) SUBFAIL ex0.py::T::test_foo - AssertionError: 4 not less than 3
(i=5) SUBFAIL ex0.py::T::test_foo - AssertionError: 5 not less than 3
(i=6) SUBFAIL ex0.py::T::test_foo - AssertionError: 6 not less than 3
=================================== 4 failed, 1 passed in 0.11s ===================================

$ python -m pytest ex1.py
...
(i=6) SUBFAIL ex1.py::T::test_foo - AssertionError: 6 not less than 3
============================= 1 failed, 1 passed, 3 skipped in 0.11s ===

• No pytest-subtests:

$ python -m pytest ex0.py
FAILED ex0.py::T::test_foo - AssertionError: 3 not less than 3
======================================== 1 failed in 0.11s ========================================

$ python -m pytest ex1.py
...
======================================= 1 skipped in 0.07s ========================================

CC: @ydshieh @ArthurZucker

[ydshieh]

It turns out this is related to pytest interacting with unittest and subTest/skipTest.

If running with python3 ex1.py (or ex0.py) with those files with ending

if __name__ == "__main__":
    unittest.main()

then subTest/skipTest works well together ...

[dvrogozh]

There are couple open issues which probably worth mentioning:

• https://github.com/pytest-dev/pytest/issues/1367 - it's opened, long thread, last update in 2021
• https://github.com/pytest-dev/pytest-subtests/issues/5 - it's also opened, it partially describes concerns we have, last commented 2023, no replies from maintainers