Test failures with pytest 8.0.2

[mweinelt]

Hi!

After updating pytest to 8.x we are seeing test failures with pytest-snapshot 0.9.0 over in nixpkgs.

============================= test session starts ==============================
platform linux -- Python 3.11.8, pytest-8.0.2, pluggy-1.4.0
rootdir: /build/source
plugins: snapshot-0.9.0
collected 108 items                                                            

tests/test_assert_match.py ....F.............................x....       [ 36%]
tests/test_assert_match_dir.py ............................              [ 62%]
tests/test_misc.py .........................................             [100%]

=================================== FAILURES ===================================
_______________________ test_assert_match_failure_bytes ________________________

request = <FixtureRequest for <Function test_assert_match_failure_bytes>>
testdir = <Testdir local('/build/pytest-of-nixbld/pytest-0/test_assert_match_failure_bytes0')>
basic_case_dir = local('/build/pytest-of-nixbld/pytest-0/test_assert_match_failure_bytes0/case_dir')

    def test_assert_match_failure_bytes(request, testdir, basic_case_dir):
        testdir.makepyfile(r"""
            import os
            def test_sth(snapshot):
                snapshot.snapshot_dir = 'case_dir'
                snapshot.assert_match(b'the INCORRECT value of snapshot1.txt' + os.linesep.encode(), 'snapshot1.txt')
        """)
        result = runpytest_with_assert_mode(testdir, request, '-v', '--assert=rewrite')
>       result.stdout.fnmatch_lines([
            r'*::test_sth FAILED*',
            r">* snapshot.assert_match(b'the INCORRECT value of snapshot1.txt' + os.linesep.encode(), 'snapshot1.txt')",
            r'E* AssertionError: value does not match the expected value in snapshot case_dir?snapshot1.txt',
            r'E*   (run pytest with --snapshot-update to update snapshots)',
            r"E* assert * == *",
            r"E* At index 4 diff: * != *",
            r"E* Full diff:",
            r"E* - b'the valu\xc3\x89 of snapshot1.txt{}'".format(repr(os.linesep)[1:-1]),
            r"E* + b'the INCORRECT value of snapshot1.txt{}'".format(repr(os.linesep)[1:-1]),
        ])
E       Failed: nomatch: '*::test_sth FAILED*'
E           and: '============================= test session starts =============================='
E           and: 'platform linux -- Python 3.11.8, pytest-8.0.2, pluggy-1.4.0 -- /nix/store/vhlk4r8ralh744khcy3vy47ijw5rvpw5-python3-3.1
E           and: 'cachedir: .pytest_cache'
E           and: 'rootdir: /build/pytest-of-nixbld/pytest-0/test_assert_match_failure_bytes0'
E           and: 'plugins: snapshot-0.9.0'
E           and: 'collecting ... collected 1 item'
E           and: ''
E       fnmatch: '*::test_sth FAILED*'
E          with: 'test_assert_match_failure_bytes.py::test_sth FAILED                      [100%]'
E       nomatch: ">* snapshot.assert_match(b'the INCORRECT value of snapshot1.txt' + os.linesep.encode(), 'snapshot1.txt')"
E           and: ''
E           and: '=================================== FAILURES ==================================='
E           and: '___________________________________ test_sth ___________________________________'
E           and: ''
E           and: 'snapshot = <pytest_snapshot.plugin.Snapshot object at 0x7ffff5e1c2d0>'
E           and: ''
E           and: '    def test_sth(snapshot):'
E           and: "        snapshot.snapshot_dir = 'case_dir'"
E       fnmatch: ">* snapshot.assert_match(b'the INCORRECT value of snapshot1.txt' + os.linesep.encode(), 'snapshot1.txt')"
E          with: ">       snapshot.assert_match(b'the INCORRECT value of snapshot1.txt' + os.linesep.encode(), 'snapshot1.txt')"
E       fnmatch: 'E* AssertionError: value does not match the expected value in snapshot case_dir?snapshot1.txt'
E          with: 'E       AssertionError: value does not match the expected value in snapshot case_dir/snapshot1.txt'
E       fnmatch: 'E*   (run pytest with --snapshot-update to update snapshots)'
E          with: 'E         (run pytest with --snapshot-update to update snapshots)'
E       fnmatch: 'E* assert * == *'
E          with: "E       assert b'the INCORRE...apshot1.txt\\n' == b'the valu\\xc...apshot1.txt\\n'"
E       nomatch: 'E* At index 4 diff: * != *'
E           and: 'E         '
E       fnmatch: 'E* At index 4 diff: * != *'
E          with: "E         At index 4 diff: b'I' != b'v'"
E       nomatch: 'E* Full diff:'
E           and: 'E         '
E       fnmatch: 'E* Full diff:'
E          with: 'E         Full diff:'
E       nomatch: "E* - b'the valu\\xc3\\x89 of snapshot1.txt\\n'"
E           and: "E         - (b'the valu\\xc3\\x89 of snapshot1.txt\\n')"
E           and: 'E         ?            ^^^^^^^^'
E           and: "E         + (b'the INCORRECT value of snapshot1.txt\\n')"
E           and: 'E         ?        ++++++++++    ^'
E           and: ''
E           and: 'test_assert_match_failure_bytes.py:4: AssertionError'
E           and: '=========================== short test summary info ============================'
E           and: 'FAILED test_assert_match_failure_bytes.py::test_sth - AssertionError: value d...'
E           and: '============================== 1 failed in 0.01s ==============================='
E       remains unmatched: "E* - b'the valu\\xc3\\x89 of snapshot1.txt\\n'"

/build/source/tests/test_assert_match.py:82: Failed
----------------------------- Captured stdout call -----------------------------
============================= test session starts ==============================
platform linux -- Python 3.11.8, pytest-8.0.2, pluggy-1.4.0 -- /nix/store/vhlk4r8ralh744khcy3vy47ijw5rvpw5-python3-3.11.8/bin/python3.11
cachedir: .pytest_cache
rootdir: /build/pytest-of-nixbld/pytest-0/test_assert_match_failure_bytes0
plugins: snapshot-0.9.0
collecting ... collected 1 item

test_assert_match_failure_bytes.py::test_sth FAILED                      [100%]

=================================== FAILURES ===================================
___________________________________ test_sth ___________________________________

snapshot = <pytest_snapshot.plugin.Snapshot object at 0x7ffff5e1c2d0>

    def test_sth(snapshot):
        snapshot.snapshot_dir = 'case_dir'
>       snapshot.assert_match(b'the INCORRECT value of snapshot1.txt' + os.linesep.encode(), 'snapshot1.txt')
E       AssertionError: value does not match the expected value in snapshot case_dir/snapshot1.txt
E         (run pytest with --snapshot-update to update snapshots)
E       assert b'the INCORRE...apshot1.txt\n' == b'the valu\xc...apshot1.txt\n'
E         
E         At index 4 diff: b'I' != b'v'
E         
E         Full diff:
E         - (b'the valu\xc3\x89 of snapshot1.txt\n')
E         ?            ^^^^^^^^
E         + (b'the INCORRECT value of snapshot1.txt\n')
E         ?        ++++++++++    ^

test_assert_match_failure_bytes.py:4: AssertionError
=========================== short test summary info ============================
FAILED test_assert_match_failure_bytes.py::test_sth - AssertionError: value d...
============================== 1 failed in 0.01s ===============================
=========================== short test summary info ============================
FAILED tests/test_assert_match.py::test_assert_match_failure_bytes - Failed: nomatch: '*::test_sth FAILED*'
=================== 1 failed, 106 passed, 1 xfailed in 4.55s ===================

[kloczek]

Similar with pytest 8.1.1

Here is pytest output:

```console + PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-pytest-snapshot-0.9.0-2.fc36.x86_64/usr/lib64/python3.9/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-pytest-snapshot-0.9.0-2.fc36.x86_64/usr/lib/python3.9/site-packages + /usr/bin/pytest -ra -m 'not network' ==================================================================================== test session starts ==================================================================================== platform linux -- Python 3.9.18, pytest-8.1.1, pluggy-1.4.0 rootdir: /home/tkloczko/rpmbuild/BUILD/pytest-snapshot-0.9.0 configfile: pyproject.toml plugins: snapshot-0.9.0 collected 108 items tests/test_assert_match.py ....F.............................x.... [ 36%] tests/test_assert_match_dir.py ............................ [ 62%] tests/test_misc.py ......................................... [100%] ========================================================================================= FAILURES ========================================================================================== ______________________________________________________________________________ test_assert_match_failure_bytes ______________________________________________________________________________ request = >, testdir = basic_case_dir = local('/tmp/pytest-of-tkloczko/pytest-30/test_assert_match_failure_bytes0/case_dir') def test_assert_match_failure_bytes(request, testdir, basic_case_dir): testdir.makepyfile(r""" import os def test_sth(snapshot): snapshot.snapshot_dir = 'case_dir' snapshot.assert_match(b'the INCORRECT value of snapshot1.txt' + os.linesep.encode(), 'snapshot1.txt') """) result = runpytest_with_assert_mode(testdir, request, '-v', '--assert=rewrite') > result.stdout.fnmatch_lines([ r'*::test_sth FAILED*', r">* snapshot.assert_match(b'the INCORRECT value of snapshot1.txt' + os.linesep.encode(), 'snapshot1.txt')", r'E* AssertionError: value does not match the expected value in snapshot case_dir?snapshot1.txt', r'E* (run pytest with --snapshot-update to update snapshots)', r"E* assert * == *", r"E* At index 4 diff: * != *", r"E* Full diff:", fr"E* - b'the valu\xc3\x89 of snapshot1.txt{repr(os.linesep)[1:-1]}'", fr"E* + b'the INCORRECT value of snapshot1.txt{repr(os.linesep)[1:-1]}'", ]) E Failed: nomatch: '*::test_sth FAILED*' E and: '============================= test session starts ==============================' E and: 'platform linux -- Python 3.9.18, pytest-8.1.1, pluggy-1.4.0 -- /usr/bin/python3' E and: 'cachedir: .pytest_cache' E and: 'rootdir: /tmp/pytest-of-tkloczko/pytest-30/test_assert_match_failure_bytes0' E and: 'plugins: snapshot-0.9.0' E and: 'collecting ... collected 1 item' E and: '' E fnmatch: '*::test_sth FAILED*' E with: 'test_assert_match_failure_bytes.py::test_sth FAILED [100%]' E nomatch: ">* snapshot.assert_match(b'the INCORRECT value of snapshot1.txt' + os.linesep.encode(), 'snapshot1.txt')" E and: '' E and: '=================================== FAILURES ===================================' E and: '___________________________________ test_sth ___________________________________' E and: '' E and: 'snapshot = ' E and: '' E and: ' def test_sth(snapshot):' E and: " snapshot.snapshot_dir = 'case_dir'" E fnmatch: ">* snapshot.assert_match(b'the INCORRECT value of snapshot1.txt' + os.linesep.encode(), 'snapshot1.txt')" E with: "> snapshot.assert_match(b'the INCORRECT value of snapshot1.txt' + os.linesep.encode(), 'snapshot1.txt')" E fnmatch: 'E* AssertionError: value does not match the expected value in snapshot case_dir?snapshot1.txt' E with: 'E AssertionError: value does not match the expected value in snapshot case_dir/snapshot1.txt' E fnmatch: 'E* (run pytest with --snapshot-update to update snapshots)' E with: 'E (run pytest with --snapshot-update to update snapshots)' E fnmatch: 'E* assert * == *' E with: "E assert b'the INCORRE...apshot1.txt\\n' == b'the valu\\xc...apshot1.txt\\n'" E nomatch: 'E* At index 4 diff: * != *' E and: 'E ' E fnmatch: 'E* At index 4 diff: * != *' E with: "E At index 4 diff: b'I' != b'v'" E nomatch: 'E* Full diff:' E and: 'E ' E fnmatch: 'E* Full diff:' E with: 'E Full diff:' E nomatch: "E* - b'the valu\\xc3\\x89 of snapshot1.txt\\n'" E and: "E - (b'the valu\\xc3\\x89 of snapshot1.txt\\n')" E and: 'E ? ^^^^^^^^' E and: "E + (b'the INCORRECT value of snapshot1.txt\\n')" E and: 'E ? ++++++++++ ^' E and: '' E and: 'test_assert_match_failure_bytes.py:4: AssertionError' E and: '=========================== short test summary info ============================' E and: 'FAILED test_assert_match_failure_bytes.py::test_sth - AssertionError: value d...' E and: '============================== 1 failed in 0.01s ===============================' E remains unmatched: "E* - b'the valu\\xc3\\x89 of snapshot1.txt\\n'" /home/tkloczko/rpmbuild/BUILD/pytest-snapshot-0.9.0/tests/test_assert_match.py:82: Failed ----------------------------------------------------------------------------------- Captured stdout call ------------------------------------------------------------------------------------ ============================= test session starts ============================== platform linux -- Python 3.9.18, pytest-8.1.1, pluggy-1.4.0 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /tmp/pytest-of-tkloczko/pytest-30/test_assert_match_failure_bytes0 plugins: snapshot-0.9.0 collecting ... collected 1 item test_assert_match_failure_bytes.py::test_sth FAILED [100%] =================================== FAILURES =================================== ___________________________________ test_sth ___________________________________ snapshot = def test_sth(snapshot): snapshot.snapshot_dir = 'case_dir' > snapshot.assert_match(b'the INCORRECT value of snapshot1.txt' + os.linesep.encode(), 'snapshot1.txt') E AssertionError: value does not match the expected value in snapshot case_dir/snapshot1.txt E (run pytest with --snapshot-update to update snapshots) E assert b'the INCORRE...apshot1.txt\n' == b'the valu\xc...apshot1.txt\n' E E At index 4 diff: b'I' != b'v' E E Full diff: E - (b'the valu\xc3\x89 of snapshot1.txt\n') E ? ^^^^^^^^ E + (b'the INCORRECT value of snapshot1.txt\n') E ? ++++++++++ ^ test_assert_match_failure_bytes.py:4: AssertionError =========================== short test summary info ============================ FAILED test_assert_match_failure_bytes.py::test_sth - AssertionError: value d... ============================== 1 failed in 0.01s =============================== ========================================================================================= XFAILURES ========================================================================================= ___________________________________________________________________________ test_assert_match_edge_cases[slash-r] ___________________________________________________________________________ testdir = basic_case_dir = local('/tmp/pytest-of-tkloczko/pytest-30/test_assert_match_edge_cases5/case_dir'), tested_value = '\r' @pytest.mark.parametrize('tested_value', [ b'', '', bytes(bytearray(range(256))), ''.join(chr(i) for i in range(0, 10000)).replace('\r', ''), ' \n \t \n Whitespace! \n\t Whitespace! \n \t \n ', # We don't support \r due to cross-compatibility and git by default modifying snapshot files... pytest.param('\r', marks=pytest.mark.xfail(strict=True)), ], ids=[ 'empty-bytes', 'empty-string', 'all-bytes', 'unicode', 'whitespace', 'slash-r', ]) def test_assert_match_edge_cases(testdir, basic_case_dir, tested_value): """ This test tests many possible values to snapshot test. This test will fail if we change the snapshot file format in any way. This test also checks that assert_match will pass after a snapshot update. """ testdir.makepyfile(r""" def test_sth(snapshot): tested_value = {tested_value!r} snapshot.snapshot_dir = 'case_dir' snapshot.assert_match(tested_value, 'tested_value_snapshot') """.format(tested_value=tested_value)) result = testdir.runpytest('-v', '--snapshot-update') > result.stdout.fnmatch_lines([ '*::test_sth PASSED*', '*::test_sth ERROR*', ]) E Failed: nomatch: '*::test_sth PASSED*' E and: '============================= test session starts ==============================' E and: 'platform linux -- Python 3.9.18, pytest-8.1.1, pluggy-1.4.0 -- /usr/bin/python3' E and: 'cachedir: .pytest_cache' E and: 'rootdir: /tmp/pytest-of-tkloczko/pytest-30/test_assert_match_edge_cases5' E and: 'plugins: snapshot-0.9.0' E and: 'collecting ... collected 1 item' E and: '' E and: 'test_assert_match_edge_cases.py::test_sth FAILED [100%]' E and: '' E and: '=================================== FAILURES ===================================' E and: '___________________________________ test_sth ___________________________________' E and: '' E and: 'snapshot = ' E and: '' E and: ' def test_sth(snapshot):' E and: " tested_value = '\\r'" E and: " snapshot.snapshot_dir = 'case_dir'" E and: "> snapshot.assert_match(tested_value, 'tested_value_snapshot')" E and: '' E and: 'test_assert_match_edge_cases.py:4: ' E and: '_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ' E and: '/home/tkloczko/rpmbuild/BUILDROOT/python-pytest-snapshot-0.9.0-2.fc36.x86_64/usr/lib/python3.9/site-packages/pytest_snapshot/plugin.py:173: in assert_match' E and: ' encoded_value = encode(value)' E and: '_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ' E and: '' E and: "string = '\\r'" E and: '' E and: ' def _file_encode(string: str) -> bytes:' E and: ' """' E and: ' Returns the bytes that would be in a file created using ``path.write_text(string)``.' E and: ' See universal newlines documentation.' E and: ' """' E and: " if '\\r' in string:" E and: "> raise ValueError('''\\" E and: ' Snapshot testing strings containing "\\\\r" is not supported.' E and: ' To snapshot test non-standard newlines you should convert the tested value to bytes.' E and: ' Warning: git may decide to modify the newlines in the snapshot file.' E and: ' To avoid this read \\' E and: " https://docs.github.com/en/get-started/getting-started-with-git/configuring-git-to-handle-line-endings''')" E and: 'E ValueError: Snapshot testing strings containing "\\r" is not supported.' E and: 'E To snapshot test non-standard newlines you should convert the tested value to bytes.' E and: 'E Warning: git may decide to modify the newlines in the snapshot file.' E and: 'E To avoid this read https://docs.github.com/en/get-started/getting-started-with-git/configuring-git-to-handle-line-endings' E and: '' E and: '/home/tkloczko/rpmbuild/BUILDROOT/python-pytest-snapshot-0.9.0-2.fc36.x86_64/usr/lib/python3.9/site-packages/pytest_snapshot/plugin.py:52: ValueError' E and: '=========================== short test summary info ============================' E and: 'FAILED test_assert_match_edge_cases.py::test_sth - ValueError: Snapshot testi...' E and: '============================== 1 failed in 0.01s ===============================' E remains unmatched: '*::test_sth PASSED*' /home/tkloczko/rpmbuild/BUILD/pytest-snapshot-0.9.0/tests/test_assert_match.py:331: Failed ----------------------------------------------------------------------------------- Captured stdout call ------------------------------------------------------------------------------------ ============================= test session starts ============================== platform linux -- Python 3.9.18, pytest-8.1.1, pluggy-1.4.0 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /tmp/pytest-of-tkloczko/pytest-30/test_assert_match_edge_cases5 plugins: snapshot-0.9.0 collecting ... collected 1 item test_assert_match_edge_cases.py::test_sth FAILED [100%] =================================== FAILURES =================================== ___________________________________ test_sth ___________________________________ snapshot = def test_sth(snapshot): tested_value = '\r' snapshot.snapshot_dir = 'case_dir' > snapshot.assert_match(tested_value, 'tested_value_snapshot') test_assert_match_edge_cases.py:4: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /home/tkloczko/rpmbuild/BUILDROOT/python-pytest-snapshot-0.9.0-2.fc36.x86_64/usr/lib/python3.9/site-packages/pytest_snapshot/plugin.py:173: in assert_match encoded_value = encode(value) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ string = '\r' def _file_encode(string: str) -> bytes: """ Returns the bytes that would be in a file created using ``path.write_text(string)``. See universal newlines documentation. """ if '\r' in string: > raise ValueError('''\ Snapshot testing strings containing "\\r" is not supported. To snapshot test non-standard newlines you should convert the tested value to bytes. Warning: git may decide to modify the newlines in the snapshot file. To avoid this read \ https://docs.github.com/en/get-started/getting-started-with-git/configuring-git-to-handle-line-endings''') E ValueError: Snapshot testing strings containing "\r" is not supported. E To snapshot test non-standard newlines you should convert the tested value to bytes. E Warning: git may decide to modify the newlines in the snapshot file. E To avoid this read https://docs.github.com/en/get-started/getting-started-with-git/configuring-git-to-handle-line-endings /home/tkloczko/rpmbuild/BUILDROOT/python-pytest-snapshot-0.9.0-2.fc36.x86_64/usr/lib/python3.9/site-packages/pytest_snapshot/plugin.py:52: ValueError =========================== short test summary info ============================ FAILED test_assert_match_edge_cases.py::test_sth - ValueError: Snapshot testi... ============================== 1 failed in 0.01s =============================== ================================================================================== short test summary info ================================================================================== XFAIL tests/test_assert_match.py::test_assert_match_edge_cases[slash-r] FAILED tests/test_assert_match.py::test_assert_match_failure_bytes - Failed: nomatch: '*::test_sth FAILED*' ========================================================================= 1 failed, 106 passed, 1 xfailed in 7.51s ========================================================================== ```

[R-Peleg]

Reproduced on pytest 8.2.2

[R-Peleg]

Seems like PyTest added '(' to its message

E       nomatch: "E* - b'the valu\\xc3\\x89 of snapshot1.txt\\r\\n'"
E           and: "E         - (b'the valu\\xc3\\x89 of snapshot1.txt\\r\\n')"