search

syrupy-project / syrupy

:pancakes: The sweeter pytest snapshot plugin
https://syrupy-project.github.io/syrupy/
Apache License 2.0
553 stars 36 forks source link

syrupy 4.6.3+ is incompatible with pytest-rerunfailures #879

Closed jcheng5 closed 3 months ago

jcheng5 commented 3 months ago

Describe the bug

Starting with syrupy 4.6.3, tests that are marked with @pytest.mark.flaky for pytest-rerunfailures cause internal error ValueError: 'rerun' is not a valid ItemStatus. The error happens on this line, which was introduced in https://github.com/syrupy-project/syrupy/commit/3f6e30182b3aed34687782385c16001c7cf0b5f3.

This happens because pytest-rerunfailures sets the outcome to the value "rerun" on this line.

To reproduce

pip install pytest pytest-rerunfailures syrupy

test_example.py:

import pytest

@pytest.mark.flaky(reruns=1, delay=0)
def test_failing():
    assert False

The outcome you get with 4.6.3 is:

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/jcheng/Development/scratchpad/syrupy-reprex/.venv/lib/python3.10/site-packages/_pytest/main.py", line 283, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/Users/jcheng/Development/scratchpad/syrupy-reprex/.venv/lib/python3.10/site-packages/_pytest/main.py", line 337, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/Users/jcheng/Development/scratchpad/syrupy-reprex/.venv/lib/python3.10/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>   File "/Users/jcheng/Development/scratchpad/syrupy-reprex/.venv/lib/python3.10/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/Users/jcheng/Development/scratchpad/syrupy-reprex/.venv/lib/python3.10/site-packages/pluggy/_callers.py", line 139, in _multicall
INTERNALERROR>     raise exception.with_traceback(exception.__traceback__)
INTERNALERROR>   File "/Users/jcheng/Development/scratchpad/syrupy-reprex/.venv/lib/python3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR>   File "/Users/jcheng/Development/scratchpad/syrupy-reprex/.venv/lib/python3.10/site-packages/_pytest/logging.py", line 805, in pytest_runtestloop
INTERNALERROR>     return (yield)  # Run all the tests.
INTERNALERROR>   File "/Users/jcheng/Development/scratchpad/syrupy-reprex/.venv/lib/python3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR>   File "/Users/jcheng/Development/scratchpad/syrupy-reprex/.venv/lib/python3.10/site-packages/_pytest/terminal.py", line 673, in pytest_runtestloop
INTERNALERROR>     result = yield
INTERNALERROR>   File "/Users/jcheng/Development/scratchpad/syrupy-reprex/.venv/lib/python3.10/site-packages/pluggy/_callers.py", line 103, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/jcheng/Development/scratchpad/syrupy-reprex/.venv/lib/python3.10/site-packages/_pytest/main.py", line 362, in pytest_runtestloop
INTERNALERROR>     item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR>   File "/Users/jcheng/Development/scratchpad/syrupy-reprex/.venv/lib/python3.10/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>   File "/Users/jcheng/Development/scratchpad/syrupy-reprex/.venv/lib/python3.10/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/Users/jcheng/Development/scratchpad/syrupy-reprex/.venv/lib/python3.10/site-packages/pluggy/_callers.py", line 139, in _multicall
INTERNALERROR>     raise exception.with_traceback(exception.__traceback__)
INTERNALERROR>   File "/Users/jcheng/Development/scratchpad/syrupy-reprex/.venv/lib/python3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR>   File "/Users/jcheng/Development/scratchpad/syrupy-reprex/.venv/lib/python3.10/site-packages/_pytest/warnings.py", line 112, in pytest_runtest_protocol
INTERNALERROR>     return (yield)
INTERNALERROR>   File "/Users/jcheng/Development/scratchpad/syrupy-reprex/.venv/lib/python3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR>   File "/Users/jcheng/Development/scratchpad/syrupy-reprex/.venv/lib/python3.10/site-packages/_pytest/assertion/__init__.py", line 176, in pytest_runtest_protocol
INTERNALERROR>     return (yield)
INTERNALERROR>   File "/Users/jcheng/Development/scratchpad/syrupy-reprex/.venv/lib/python3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR>   File "/Users/jcheng/Development/scratchpad/syrupy-reprex/.venv/lib/python3.10/site-packages/_pytest/unittest.py", line 429, in pytest_runtest_protocol
INTERNALERROR>     res = yield
INTERNALERROR>   File "/Users/jcheng/Development/scratchpad/syrupy-reprex/.venv/lib/python3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR>   File "/Users/jcheng/Development/scratchpad/syrupy-reprex/.venv/lib/python3.10/site-packages/_pytest/faulthandler.py", line 87, in pytest_runtest_protocol
INTERNALERROR>     return (yield)
INTERNALERROR>   File "/Users/jcheng/Development/scratchpad/syrupy-reprex/.venv/lib/python3.10/site-packages/pluggy/_callers.py", line 103, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/jcheng/Development/scratchpad/syrupy-reprex/.venv/lib/python3.10/site-packages/pytest_rerunfailures.py", line 563, in pytest_runtest_protocol
INTERNALERROR>     item.ihook.pytest_runtest_logreport(report=report)
INTERNALERROR>   File "/Users/jcheng/Development/scratchpad/syrupy-reprex/.venv/lib/python3.10/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>   File "/Users/jcheng/Development/scratchpad/syrupy-reprex/.venv/lib/python3.10/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/Users/jcheng/Development/scratchpad/syrupy-reprex/.venv/lib/python3.10/site-packages/pluggy/_callers.py", line 139, in _multicall
INTERNALERROR>     raise exception.with_traceback(exception.__traceback__)
INTERNALERROR>   File "/Users/jcheng/Development/scratchpad/syrupy-reprex/.venv/lib/python3.10/site-packages/pluggy/_callers.py", line 103, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/jcheng/Development/scratchpad/syrupy-reprex/.venv/lib/python3.10/site-packages/syrupy/__init__.py", line 157, in pytest_runtest_logreport
INTERNALERROR>     _syrupy.ran_item(report.nodeid, report.outcome)
INTERNALERROR>   File "/Users/jcheng/Development/scratchpad/syrupy-reprex/.venv/lib/python3.10/site-packages/syrupy/session.py", line 125, in ran_item
INTERNALERROR>     self._selected_items[nodeid] = ItemStatus(outcome)
INTERNALERROR>   File "/Users/jcheng/.pyenv/versions/3.10.6/lib/python3.10/enum.py", line 385, in __call__
INTERNALERROR>     return cls.__new__(cls, value)
INTERNALERROR>   File "/Users/jcheng/.pyenv/versions/3.10.6/lib/python3.10/enum.py", line 710, in __new__
INTERNALERROR>     raise ve_exc
INTERNALERROR> ValueError: 'rerun' is not a valid ItemStatus

====================================================================================== 1 rerun in 0.02s ======================================================================================

Expected behavior

The desired output looks like this (using syrupy 4.6.2):

========================================================================================== FAILURES ==========================================================================================
________________________________________________________________________________________ test_failing ________________________________________________________________________________________

    @pytest.mark.flaky(reruns=1, delay=0)
    def test_failing():
>       assert False
E       assert False

test_example.py:5: AssertionError
================================================================================== short test summary info ===================================================================================
FAILED test_example.py::test_failing - assert False
================================================================================= 1 failed, 1 rerun in 0.02s =================================================================================

Environment (please complete the following information):

noahnu commented 3 months ago

:tada: This issue has been resolved in version 4.7.1 :tada:

The release is available on:

Your semantic-release bot :package::rocket:

noahnu commented 3 months ago

Let me know if 4.7.1 fixes the issue for you. I've tested it manually and seems to work.

I need to follow up on how to properly test this to prevent regressions.. for other plugins we sometimes mimic the behaviour of the plugin in a test. So in this case we can create a plugin within a test that sets the outcome to some arbitrary value.

Longer term, I'd like to add compatibility tests for a number of the popular plugins out there. Created #882 to track that

jcheng5 commented 3 months ago

Works great—thank you for the amazingly fast turnaround!