Test failures

[fabaff]

In NixOS we get some test failures. It happens on Python 3.10 too. At first glances I would say it's pytest-related.

platform linux -- Python 3.11.4, pytest-7.4.0, pluggy-1.2.0
rootdir: /build/source
configfile: setup.cfg
plugins: raises-0.11
collected 30 items                                                             

tests/test_raises.py ....F......F.F................                      [100%]

=================================== FAILURES ===================================
_________________ test_pytest_mark_raises_unexpected_exception _________________

testdir = <Testdir local('/build/pytest-of-nixbld/pytest-0/test_pytest_mark_raises_unexpected_exception0')>

    def test_pytest_mark_raises_unexpected_exception(testdir):
>       _run_tests_test(testdir, """
                import pytest

                class SomeException(Exception):
                    pass

                class AnotherException(Exception):
                    pass

                @pytest.mark.raises(exception = SomeException)
                def test_pytest_mark_raises_unexpected_exception():
                    raise AnotherException('the message')
            """,
            [
                '*::test_pytest_mark_raises_unexpected_exception FAILED*',
                # pylint: disable=line-too-long
                "*ExpectedException: Expected exception of type <class '*SomeException'>, but got exception of type <class '*AnotherException'> with message: the message",
            ],
            1
        )

/build/source/tests/test_raises.py:83: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

testdir = <Testdir local('/build/pytest-of-nixbld/pytest-0/test_pytest_mark_raises_unexpected_exception0')>
code = "\n            import pytest\n\n            class SomeException(Exception):\n                pass\n\n            class...  def test_pytest_mark_raises_unexpected_exception():\n                raise AnotherException('the message')\n        "
expected_output = ['*::test_pytest_mark_raises_unexpected_exception FAILED*', "*ExpectedException: Expected exception of type <class '*SomeException'>, but got exception of type <class '*AnotherException'> with message: the message"]
expcted_return_code = 1, conftest = None

    def _run_tests_test(testdir, code, expected_output, expcted_return_code, conftest=None):
        if conftest:
            testdir.makeconftest(conftest)
        testdir.makepyfile(code)
        result = testdir.runpytest('-v')
>       result.stdout.fnmatch_lines(expected_output)
E       Failed: nomatch: '*::test_pytest_mark_raises_unexpected_exception FAILED*'
E           and: '============================= test session starts =============================='
E           and: 'platform linux -- Python 3.11.4, pytest-7.4.0, pluggy-1.2.0 -- /nix/store/9hwaihb8rhlvq8s7xq11m204jp51h6xi-python3-3.11.4/bin/python3.11'
E           and: 'cachedir: .pytest_cache'
E           and: 'rootdir: /build/pytest-of-nixbld/pytest-0/test_pytest_mark_raises_unexpected_exception0'
E           and: 'plugins: raises-0.11'
E           and: 'collecting ... collected 1 item'
E           and: ''
E       fnmatch: '*::test_pytest_mark_raises_unexpected_exception FAILED*'
E          with: 'test_pytest_mark_raises_unexpected_exception.py::test_pytest_mark_raises_unexpected_exception FAILED [100%]'
E       nomatch: "*ExpectedException: Expected exception of type <class '*SomeException'>, but got exception of type <class '*AnotherException'> with message: the message"
E           and: ''
E           and: '=================================== FAILURES ==================================='
E           and: '_________________ test_pytest_mark_raises_unexpected_exception _________________'
E           and: "Expected exception of type <class 'test_pytest_mark_raises_unexpected_exception.SomeException'>, but got exception of type <class 'test_pytest_mark_raises_unexpected_exception.AnotherException'> with message: the message"
E           and: 'All traceback entries are hidden. Pass `--full-trace` to see hidden and internal frames.'
E           and: ''
E           and: 'During handling of the above exception, another exception occurred:'
E           and: "property 'excinfo' of '_Result' object has no setter"
E           and: 'All traceback entries are hidden. Pass `--full-trace` to see hidden and internal frames.'
E           and: ''
E           and: 'During handling of the above exception, another exception occurred:'
E           and: "'_Result' object has no attribute '_excinfo'"
E           and: 'All traceback entries are hidden. Pass `--full-trace` to see hidden and internal frames.'
E           and: ''
E           and: 'During handling of the above exception, another exception occurred:'
E           and: "Expected exception of type <class 'test_pytest_mark_raises_unexpected_exception.SomeException'>, but got exception of type <class 'test_pytest_mark_raises_unexpected_exception.AnotherException'> with message: the message"
E           and: 'All traceback entries are hidden. Pass `--full-trace` to see hidden and internal frames.'
E           and: '=========================== short test summary info ============================'
E           and: 'FAILED test_pytest_mark_raises_unexpected_exception.py::test_pytest_mark_raises_unexpected_exception'
E           and: '============================== 1 failed in 0.01s ==============================='
E       remains unmatched: "*ExpectedException: Expected exception of type <class '*SomeException'>, but got exception of type <class '*AnotherException'> with message: the message"

/build/source/tests/test_raises.py:8: Failed
----------------------------- Captured stdout call -----------------------------
============================= test session starts ==============================
platform linux -- Python 3.11.4, pytest-7.4.0, pluggy-1.2.0 -- /nix/store/9hwaihb8rhlvq8s7xq11m204jp51h6xi-python3-3.11.4/bin/python3.11
[...]
/build/source/tests/test_raises.py:8: Failed
----------------------------- Captured stdout call -----------------------------
============================= test session starts ==============================
platform linux -- Python 3.11.4, pytest-7.4.0, pluggy-1.2.0 -- /nix/store/9hwaihb8rhlvq8s7xq11m204jp51h6xi-python3-3.11.4/bin/python3.11
cachedir: .pytest_cache
rootdir: /build/pytest-of-nixbld/pytest-0/test_pytest_mark_raises_parametrize0
plugins: raises-0.11
collecting ... collected 9 items

test_pytest_mark_raises_parametrize.py::test_mark_raises[None0] PASSED   [ 11%]
test_pytest_mark_raises_parametrize.py::test_mark_raises[error1] PASSED  [ 22%]
test_pytest_mark_raises_parametrize.py::test_mark_raises[error2] PASSED  [ 33%]
test_pytest_mark_raises_parametrize.py::test_mark_raises[error3] PASSED  [ 44%]
test_pytest_mark_raises_parametrize.py::test_mark_raises[error4] FAILED  [ 55%]
test_pytest_mark_raises_parametrize.py::test_mark_raises[error5] FAILED  [ 66%]
test_pytest_mark_raises_parametrize.py::test_mark_raises[None1] FAILED   [ 77%]
test_pytest_mark_raises_parametrize.py::test_mark_raises[error7] PASSED  [ 88%]
test_pytest_mark_raises_parametrize.py::test_mark_raises[error8] FAILED  [100%]

=================================== FAILURES ===================================
___________________________ test_mark_raises[error4] ___________________________
Expected exception of type <class 'test_pytest_mark_raises_parametrize.SomeException'>, but got exception of type <class 'test_pytest_mark_raises_parametrize.AnotherException'> with message: the message
All traceback entries are hidden. Pass `--full-trace` to see hidden and internal frames.

During handling of the above exception, another exception occurred:
property 'excinfo' of '_Result' object has no setter
All traceback entries are hidden. Pass `--full-trace` to see hidden and internal frames.

During handling of the above exception, another exception occurred:
'_Result' object has no attribute '_excinfo'
All traceback entries are hidden. Pass `--full-trace` to see hidden and internal frames.

During handling of the above exception, another exception occurred:
Expected exception of type <class 'test_pytest_mark_raises_parametrize.SomeException'>, but got exception of type <class 'test_pytest_mark_raises_parametrize.AnotherException'> with message: the message
All traceback entries are hidden. Pass `--full-trace` to see hidden and internal frames.
___________________________ test_mark_raises[error5] ___________________________

error = SomeException('the message')

    @pytest.mark.parametrize('error', [
        None,
        pytest.param(
            SomeException('the message'), marks=pytest.mark.raises(exception=SomeException)
        ),
        pytest.param(
            AnotherException('the message'), marks=pytest.mark.raises(exception=AnotherException)
        ),
        pytest.param(
            Exception('the message'), marks=pytest.mark.raises()
        ),
        pytest.param(
            AnotherException('the message'), marks=pytest.mark.raises(exception=SomeException)
        ),
        SomeException('the message'),
        pytest.param(
            None, marks=pytest.mark.raises(exception=SomeException)
        ),
        pytest.param(
            SomeException('the message'), marks=pytest.mark.raises(exception=SomeException, message='the message')
        ),
        pytest.param(
            SomeException('the message'),
            marks=pytest.mark.raises(exception=SomeException, message='other message')
        ),
    ])
    def test_mark_raises(error):
        if error:
>           raise error
E           test_pytest_mark_raises_parametrize.SomeException: the message

test_pytest_mark_raises_parametrize.py:37: SomeException
___________________________ test_mark_raises[None1] ____________________________
Expected exception <class 'test_pytest_mark_raises_parametrize.SomeException'>, but it did not raise
All traceback entries are hidden. Pass `--full-trace` to see hidden and internal frames.

During handling of the above exception, another exception occurred:
property 'excinfo' of '_Result' object has no setter
All traceback entries are hidden. Pass `--full-trace` to see hidden and internal frames.

During handling of the above exception, another exception occurred:
'_Result' object has no attribute '_excinfo'
All traceback entries are hidden. Pass `--full-trace` to see hidden and internal frames.

During handling of the above exception, another exception occurred:
Expected exception <class 'test_pytest_mark_raises_parametrize.SomeException'>, but it did not raise
All traceback entries are hidden. Pass `--full-trace` to see hidden and internal frames.
___________________________ test_mark_raises[error8] ___________________________
"other message" not in "the message"
All traceback entries are hidden. Pass `--full-trace` to see hidden and internal frames.

During handling of the above exception, another exception occurred:
property 'excinfo' of '_Result' object has no setter
All traceback entries are hidden. Pass `--full-trace` to see hidden and internal frames.

During handling of the above exception, another exception occurred:
'_Result' object has no attribute '_excinfo'
All traceback entries are hidden. Pass `--full-trace` to see hidden and internal frames.

During handling of the above exception, another exception occurred:
"other message" not in "the message"
All traceback entries are hidden. Pass `--full-trace` to see hidden and internal frames.
=========================== short test summary info ============================
FAILED test_pytest_mark_raises_parametrize.py::test_mark_raises[error4] - Fai...
FAILED test_pytest_mark_raises_parametrize.py::test_mark_raises[error5] - tes...
FAILED test_pytest_mark_raises_parametrize.py::test_mark_raises[None1] - Fail...
FAILED test_pytest_mark_raises_parametrize.py::test_mark_raises[error8] - Fai...
========================= 4 failed, 5 passed in 0.02s ==========================
=========================== short test summary info ============================
FAILED tests/test_raises.py::test_pytest_mark_raises_unexpected_exception - Failed: nomatch: '*::test_pytest_mark_raises_unexpected_exception FAILED*'
FAILED tests/test_raises.py::test_pytest_mark_raises_unexpected_match - Failed: nomatch: '*::test_pytest_mark_raises_unexpected_match FAILED*'
FAILED tests/test_raises.py::test_pytest_mark_raises_parametrize - Failed: nomatch: '*::test_mark_raises*None0* PASSED*'
========================= 3 failed, 27 passed in 1.19s =========================

[jayaddison]

The problem seems to be that the failure_message variable isn't being prefixed with the exception classname.

One of the failing tests looks for a pattern *ExpectedMessage: "other message" not in "the message" -- but what appears (as shown in your output sample) is only the "other message" not in "the message" part.

The exception classname prefix is added in one of two places, within the same method: either here or here.

I guess that means that the code has begun following this path instead in the affected tests.. but I'm not sure why that is yet.

[jayaddison]

NB: also, some of the excinfo output warnings appear to be related to the removal of the _Result._excinfo attribute in pluggy v1.1.0 onwards.