Weird output for long string equality assert diffs

altendky commented 4 years ago

Good:

E       AssertionError: assert 'a\na\na\na\na\na\na\na

Bad:

E       AssertionError: assert ('a\n'\n 'a\n'\n 'a\n'\n

Difference: The strings being compared are one character longer in the bad case.

This came up while asserting equality of GitHub Actions config files (strs) in PyCharm and noting that it wasn't showing the handy <Click to see difference> link (https://youtrack.jetbrains.com/issue/PY-42076). To be clear, the captures below were taken from a shell so as to avoid PyCharm messing things up. The PyCharm issue happens at larger lengths than this does.

This issue was referred to in #6757 but that ticket seems focused on other aspects of the output.

altendky@p1:~/repos/ciborg$ uname -a
Linux p1 5.3.13-altendky-iss510-b2d13b5a459a4dd0c571812011d1588fa77dfea7 #1 SMP Mon Dec 2 13:22:03 EST 2019 x86_64 x86_64 x86_64 GNU/Linux

altendky@p1:~/repos/ciborg$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 19.10
Release:        19.10
Codename:       eoan

altendky@p1:~/repos/ciborg$ virtualenv -p python3.8 venv.test
created virtual environment CPython3.8.2.final.0-64 in 115ms
  creator CPython3Posix(dest=/home/altendky/repos/ciborg/venv.test, clear=False, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/home/altendky/.local/share/virtualenv/seed-app-data/v1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

altendky@p1:~/repos/ciborg$ venv.test/bin/pip install git+https://github.com/pytest-dev/pytest#egg=pytest
Collecting pytest
  Cloning https://github.com/pytest-dev/pytest to /tmp/pip-install-ozh3cks8/pytest
  Running command git clone -q https://github.com/pytest-dev/pytest /tmp/pip-install-ozh3cks8/pytest
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting attrs>=17.4.0
  Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Collecting py>=1.5.0
  Using cached py-1.8.1-py2.py3-none-any.whl (83 kB)
Collecting packaging
  Using cached packaging-20.3-py2.py3-none-any.whl (37 kB)
Collecting pluggy<1.0,>=0.12
  Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting wcwidth
  Using cached wcwidth-0.1.9-py2.py3-none-any.whl (19 kB)
Collecting more-itertools>=4.0.0
  Using cached more_itertools-8.2.0-py3-none-any.whl (43 kB)
Collecting pyparsing>=2.0.2
  Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting six
  Using cached six-1.14.0-py2.py3-none-any.whl (10 kB)
Building wheels for collected packages: pytest
  Building wheel for pytest (PEP 517) ... done
  Created wheel for pytest: filename=pytest-5.4.1.dev146+g9bf3efb8e-py3-none-any.whl size=249018 sha256=d4d4099b58b94818996ff3ade981162d6f0ada0423be1c21049233eda2928043
  Stored in directory: /tmp/pip-ephem-wheel-cache-o2ts6g20/wheels/71/a1/e6/24d12ec774c84a6b2173782c0ca120faa2de855d11ff1e6c5d
Successfully built pytest
Installing collected packages: attrs, py, pyparsing, six, packaging, pluggy, wcwidth, more-itertools, pytest
Successfully installed attrs-19.3.0 more-itertools-8.2.0 packaging-20.3 pluggy-0.13.1 py-1.8.1 pyparsing-2.4.7 pytest-5.4.1.dev146+g9bf3efb8e six-1.14.0 wcwidth-0.1.9

altendky@p1:~/repos/ciborg$ venv/bin/pip list
Package               Version                 Location                       
--------------------- ----------------------- -------------------------------
appdirs               1.4.3                   
attrs                 19.3.0                  
ciborg                0.1                     /home/altendky/repos/ciborg/src
click                 7.1.1                   
coverage              5.1                     
distlib               0.3.0                   
filelock              3.0.12                  
importlib-resources   1.4.0                   
marshmallow           3.5.1                   
marshmallow-polyfield 5.9                     
more-itertools        8.2.0                   
mypy                  0.770                   
mypy-extensions       0.4.3                   
packaging             20.3                    
pep517                0.8.2                   
pip                   20.0.2                  
pluggy                0.13.1                  
py                    1.8.1                   
pyparsing             2.4.7                   
pyrsistent            0.16.0                  
pytest                5.4.1.dev146+g9bf3efb8e 
pytest-cov            2.8.1                   
PyYAML                5.3.1                   
setuptools            45.2.0                  
six                   1.14.0                  
toml                  0.10.0                  
tox                   3.14.6                  
typed-ast             1.4.1                   
typing-extensions     3.7.4.2                 
virtualenv            20.0.18                 
wcwidth               0.1.9                   
wheel                 0.34.2

altendky@p1:~/repos/ciborg$ cat x.py
a = 'a\n'*500
b = 'b\n'*500

n = 52

def test_x():
    assert a[:n] == b[:n]

def test_y():
    assert a[:n + 1] == b[:n + 1]

altendky@p1:~/repos/ciborg$ venv.test/bin/pytest -vv x.py
====================================================================================================================== test session starts =======================================================================================================================
platform linux -- Python 3.8.2, pytest-5.4.1.dev146+g9bf3efb8e, py-1.8.1, pluggy-0.13.1 -- /home/altendky/repos/ciborg/venv.test/bin/python
cachedir: .pytest_cache
rootdir: /home/altendky/repos/ciborg, inifile: pytest.ini
collected 2 items                                                                                                                                                                                                                                                

x.py::test_x FAILED                                                                                                                                                                                                                                        [ 50%]
x.py::test_y FAILED                                                                                                                                                                                                                                        [100%]

============================================================================================================================ FAILURES ============================================================================================================================
_____________________________________________________________________________________________________________________________ test_x _____________________________________________________________________________________________________________________________

    def test_x():
>       assert a[:n] == b[:n]
E       AssertionError: assert 'a\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\n' == 'b\nb\nb\nb\nb\nb\nb\nb\nb\nb\nb\nb\nb\nb\nb\nb\nb\nb\nb\nb\nb\nb\nb\nb\nb\nb\n'
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a

x.py:8: AssertionError
_____________________________________________________________________________________________________________________________ test_y _____________________________________________________________________________________________________________________________

    def test_y():
>       assert a[:n + 1] == b[:n + 1]
E       AssertionError: assert ('a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a') == ('b\n'\n 'b\n'\n 'b\n'\n 'b\n'\n 'b\n'\n 'b\n'\n 'b\n'\n 'b\n'\n 'b\n'\n 'b\n'\n 'b\n'\n 'b\n'\n 'b\n'\n 'b\n'\n 'b\n'\n 'b\n'\n 'b\n'\n 'b\n'\n 'b\n'\n 'b\n'\n 'b\n'\n 'b\n'\n 'b\n'\n 'b\n'\n 'b\n'\n 'b\n'\n 'b')
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         - b
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a

x.py:11: AssertionError
==================================================================================================================== short test summary info =====================================================================================================================
FAILED x.py::test_x - AssertionError: assert 'a\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\n' == 'b\nb\nb\nb\nb\nb\nb\nb\nb\nb\nb\nb\nb\nb\nb\nb\nb\nb\nb\nb\nb\nb\nb\nb\nb\nb\n'
FAILED x.py::test_y - AssertionError: assert ('a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n 'a\n'\n '...
======================================================================================================================= 2 failed in 0.05s ========================================================================================================================

altendky commented 4 years ago

No fix there in the PR but a test anyways in the suite.

altendky commented 4 years ago

I traced this down to a pprint.pformat() call in _pytest._io.saferepr.safeformat(). Passing width=99999 for example improves my case as does just using repr(). I'm looking into existing tests related to this to see what the existing requirements are.

https://github.com/pytest-dev/pytest/blob/b7ad4c2bed3906e1a12d9bb1829328b4045a72de/src/_pytest/_io/saferepr.py#L64-L72

Python 3.8.2 (default, Mar  7 2020, 20:18:24) 
[GCC 9.2.1 20191008] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pprint
>>> s26 = 's\n' * 26
>>> s27 = 's\n' * 27
>>> pprint.pformat(s26)
"'s\\ns\\ns\\ns\\ns\\ns\\ns\\ns\\ns\\ns\\ns\\ns\\ns\\ns\\ns\\ns\\ns\\ns\\ns\\ns\\ns\\ns\\ns\\ns\\ns\\ns\\n'"
>>> pprint.pformat(s27)
"('s\\n'\n 's\\n'\n 's\\n'\n 's\\n'\n 's\\n'\n 's\\n'\n 's\\n'\n 's\\n'\n 's\\n'\n 's\\n'\n 's\\n'\n 's\\n'\n 's\\n'\n 's\\n'\n 's\\n'\n 's\\n'\n 's\\n'\n 's\\n'\n 's\\n'\n 's\\n'\n 's\\n'\n 's\\n'\n 's\\n'\n 's\\n'\n 's\\n'\n 's\\n'\n 's\\n')"

altendky commented 4 years ago

pprint.pformat() sorts dicts, so repr() will not match present behavior (including failing a py35 test in CI due to lack of maintaining insertion order). pprint.pformat(obj, width=1000000) won't match because then (of course...) pprint won't wrap stuff (though this does pass CI).

So, one probably bad idea would be to special case strings. Or perhaps I'm looking at the wrong thing as the error and it's not that pprint is bad here but rather that it's getting corrupted downstream such that it doesn't get presented as pprint intended. Maybe related to the newline rewrites because of other pieces I forget the names of at the moment.

And of course the situation I really care about is PyCharm's diff link so I'll have to dig into what that actually supports.

pytest-dev / pytest

Weird output for long string equality assert diffs #7127