Failed test output is less verbose with -v than not using -v

[pfctdayelise]

I don't know if this has changed recently but it seems in some cases using -v makes the output less verbose...

def test_verbosity():
    s = [{'SKU': '12345678', 'Name': 'Linda', 'Description': 'Fancy dress', 'Net Price': '9.16', 'VAT rate': '20', 'VAT': '1.83', 'Gross Price': '10.99', 'Colour': 'Grey', 'Image': 'FancyDressImage'}, {'SKU': '12345679', 'Name': 'Jane', 'Description': 'Skirt', 'Net Price': '10.00', 'VAT rate': '0', 'VAT': '0.00', 'Gross Price': '10.00', 'Colour': 'Pink', 'Image': 'ChildsSkirt'}]
    assert len(s) == 1

Normal output

$ pytest -k verbosity --tb=short
====================================== test session starts ======================================
platform linux -- Python 3.6.7, pytest-4.4.1, py-1.8.0, pluggy-0.9.0
rootdir: /mnt/c/Users/bjl/wsl/bdd-examples-project/tests, inifile: pytest.ini
plugins: bdd-3.1.0
collected 27 items / 26 deselected / 1 selected

steps/test_example.py F                                                                   [100%]

=========================================== FAILURES ============================================
________________________________________ test_verbosity _________________________________________
steps/test_example.py:55: in test_verbosity
    assert len(s) == 1
E   AssertionError: assert 2 == 1
E    +  where 2 = len([{'Colour': 'Grey', 'Description': 'Fancy dress', 'Gross Price': '10.99', 'Image': 'FancyDressImage', ...}, {'Colour': 'Pink', 'Description': 'Skirt', 'Gross Price': '10.00', 'Image': 'ChildsSkirt', ...}])
============================ 1 failed, 26 deselected in 0.12 seconds ============================

Output with -v

$ pytest -k verbosity --tb=short -v
====================================== test session starts ======================================
platform linux -- Python 3.6.7, pytest-4.4.1, py-1.8.0, pluggy-0.9.0 -- /home/bjl/.local/share/virtualenvs/bdd-examples-project-dSboamh_/bin/python3
cachedir: .pytest_cache
rootdir: /mnt/c/Users/bjl/wsl/bdd-examples-project/tests, inifile: pytest.ini
plugins: bdd-3.1.0
collected 27 items / 26 deselected / 1 selected

steps/test_example.py::test_verbosity FAILED                                              [100%]

=========================================== FAILURES ============================================
________________________________________ test_verbosity _________________________________________
steps/test_example.py:55: in test_verbosity
    assert len(s) == 1
E   assert 2 == 1
E     -2
E     +1
============================ 1 failed, 26 deselected in 0.12 seconds ============================

Calling it with higher levels of verbosity (-vv, -vvv) has the same output as -v.

Using pytest 4.4.1.

[blueyed]

I think it has not changed (but you might want to git-bisect). IIRC I've noticed this also with some PR that touched assertion rewriting.

[blueyed]

https://github.com/pytest-dev/pytest/pull/4851/files#diff-88d64a3c422417350f0a81bea982d8f6R200

[nicoddemus]

It seems it did change, tried with an old version (3.6.0) and I get the correct output with -v.

Bisected to 85055a9efecfd729e73772c9c32ff8fbba18f469, introduced by https://github.com/pytest-dev/pytest/issues/2256. cc @oscarbenjamin

[blueyed]

Some WIP at https://github.com/pytest-dev/pytest/pull/5933 (RFC).

[mdmintz]

@nicoddemus , @blueyed This is the culprit: https://github.com/pytest-dev/pytest/blob/6aaa017b1e81f6eccc48ee4f6b52d25c49747554/src/_pytest/assertion/util.py#L226-L227

        elif verbose > 0:
            explanation = _compare_eq_verbose(left, right)

That's why the failed test diff output is less verbose with -v (or -vv) than not using -v. I'd assume we just want to remove those lines so that we maintain the default behavior of having verbose diffs, which is what most people want. I also noticed that the tests are incorrect, which is why this issue wasn't caught in the first place.

[mdmintz]

I might need to revise my solution above. It'll work for an assertion such as assert len(s) == 1, where the default verbosity diff is more verbose than using -v or -vv, but that's not the same effect as assertions of other data types.

[nicoddemus]

Hi @mdmintz,

Thanks for bringing this up.

I'm opening a PR applying your suggestion above, plus another small bug that I found during my testing.

We can discuss this over the code. 👍