Open pmeier opened 2 years ago
Hi @pmeier,
Thanks for the detailed report, appreciate it!
If you have the time, please consider opening a pull request. 👍
I'll send a PR if I figure out how this can be solved. My current understanding is that first the GeneratorExit
is raised and only afterwards the actual error. Thus, I think we need more than one context manager. Whether this has to be in user code or can live in subtests.test
has yet to be determined.
FWIW, unittest.TestCase.subTest
also cannot handle this:
import unittest
class TestFoo(unittest.TestCase):
def gen(self, n):
for i in range(n):
with self.subTest(msg=str(i)):
yield i
def test_bar(self):
for i in self.gen(5):
assert i % 2 == 0
$ python -m unittest main.py
Exception ignored in: <generator object TestFoo.gen at 0x7f48747bf150>
RuntimeError: generator ignored GeneratorExit
F
======================================================================
ERROR: test_foo (main.TestFoo) [1]
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/user/main.py", line 33, in gen
yield i
GeneratorExit
======================================================================
FAIL: test_foo (main.TestFoo)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/user/main.py", line 37, in test_foo
assert i % 2 == 0
AssertionError
----------------------------------------------------------------------
Ran 1 test in 0.000s
FAILED (failures=1, errors=1)
If I'm reading this right
Raises a
GeneratorExit
at the point where the generator function was paused. [...] If the generator yields a value, aRuntimeError
is raised.
one cannot yield values after the first failure. This of course would completely defeat the purpose of a subtest.
My best guess is that we are hitting a language limitation here:
import contextlib
def gen(n):
for i in range(n):
print(f"Before yield {i}")
with contextlib.suppress(GeneratorExit):
yield i
print(f"After yield {i}")
for i in gen(3):
print(f"Processing {i}")
break
Before yield 0
Processing 0
After yield 0
Before yield 1
Exception ignored in: <generator object gen at 0x7ffb35f19550>
RuntimeError: generator ignored GeneratorExit
So, after the first GeneratorExit
was raised and even if we catch it, we can no longer yield anything new. As soon as you try, you'll get a RuntimeError
.
its certainly a language "limitation", as its a intentionally NOT supported pattern
and structurally its absolutely valid, we should go as far as letting subtest special case generator exit and taising a ProgrammingError, as structurally, the generator is intentionally disconnected from exceptions in the outer loop body, outer exception -> generator close
@nicoddemus i propose ensuring subtests transfer generator-exits as is + triggering a warning for the subtest/test
i propose ensuring subtests transfer generator-exits as is + triggering a warning for the subtest/test
I can send a PR for that. What is the preferred way to warn users from inside a pytest.plugin
? warnings.wan("foo", UserWarning)
?
Imagine I have similar test case setup for multiple tests and I want to use
subtests
:This gives the following output
As you can see, although the
subtests.test
context manager is in place, the execution stops after the first failure. Since it is aFAILED
instead of aSUBFAILED
, one also misses out on the extra information the sub failure would print.Digging into the code, the problem is
https://github.com/pytest-dev/pytest-subtests/blob/90df760933897a3089b68fe19bed0185019ee11a/pytest_subtests.py#L168-L171
not handling the
GeneratorExit
.