Open andrewasheridan opened 6 days ago
Thanks for reporting this. I don't think it has anything specifically to do with the new release, as I see the same type of behavior locally using older versions of Pytest. Just to be clear, compatibility with legacy unittest support in Pytest was never part of my scope when I designed this plugin, and I actually don't have a ton of familiarity with that particular subsystem. I do know that unittest cases function quite differently under the hood from standard Pytest cases and I suspect the issue is actually more related to this architectural difference, as indicated in the unittest compatibility page of the Pytest docs: https://docs.pytest.org/en/7.1.x/how-to/unittest.html
Due to architectural differences between the two frameworks, setup and teardown for unittest-based tests is performed during the call phase of testing instead of in pytest’s standard setup and teardown stages. This can be important to understand in some situations, particularly when reasoning about errors. For example, if a unittest-based suite exhibits errors during setup, pytest will report no errors during its setup phase and will instead raise the error during call.
With a bit of cursory debugging, it's clear that pytest-retry's current method for retrying individual tests is incompatible with the unittest implementation. As soon as I add a unittest.TestCase
inheritance to the test class, the call
object for subsequent attempts no longer includes any exception or result data regardless of whether the attempt failed, so my current solution for determining the test outcome isn't sufficient here. I'll need to do a more detailed investigation into the test runner process for unittest cases as I suspect I might need to add a separate control flow for when unittest is detected. The way you worded your report seems to imply that your unittest cases were working with an older version of Pytest and/or pytest-retry. If so, could you tell me what version that was? It would certainly help determine what changes led to the issue.
I won't make any promises here as unittest compatibility isn't particularly high on my priority list for this plugin, but I'll try to come up with an implementation as long as it doesn't compromise the design too much.
Thanks again for looking into it.
To be honest I only noticed the incorrect 'passing' messaging while typing up this issue.
Checking now I see that as far back as pytest 7.4.4 and retry 1.0.0 the test above results in "1 passed, 1 error, 1 retried", even though that test can never pass.
If changed to a not-unittest test the report is correct (1 failed, 1 retried)
So I would guess it's not due to a recent change.
All testing was on Python 3.12
Results now show a generic
instead of the actual test.
To reproduce:
test_foo.py
:Correctly shows
failed
, but assertion error points to pytest-retry code w/ pytest 8.2.2vs incorrectly shows
passed
, but correctly points to actual test code w/ pytest 8.2.1Noticed this b/c in Jenkins it just shows this:
Thanks for your attention and the helpful plugin :)