search

pytest-dev / pytest

The pytest framework makes it easy to write small tests, yet scales to support complex functional testing
https://pytest.org
MIT License
12.14k stars 2.69k forks source link

Provide dedicated control over truncation of "short test summary info" lines #10264

Open yarikoptic opened 2 years ago

yarikoptic commented 2 years ago

What's the problem this feature will solve?

Currently we cannot easily grasp the summary of underlying fails/errors which happened on CIs (logs archived using https://github.com/con/tinuous/) because "short test summary info" truncates to the width of the terminal which is in non-interactive session happens to be around 80. So we get just e.g.

(git)smaug:/mnt/datasets/datalad/ci/logs/2022/08[master]git
$> git grep  '^FAIL.*test_gracefull_dea'                               
03/pr/6915/964b1f4/travis-14378-failed/2.txt:FAILED ../datalad/support/tests/test_parallel.py::test_gracefull_death - asse...
03/pr/6915/964b1f4/travis-14378-failed/4.txt:FAILED ../support/tests/test_parallel.py::test_gracefull_death - assert 0 >= 2
23/pr/6958/fa7cee0/travis-14469-failed/9.txt:FAILED ../datalad/support/tests/test_parallel.py::test_gracefull_death - asse...
29/pr/6981/2551844/travis-14510-failed/3.txt:FAILED ../support/tests/test_parallel.py::test_gracefull_death - assert [0, 2...
30/push/maint/870c72a/travis-14519-failed/9.txt:FAILED ../datalad/support/tests/test_parallel.py::test_gracefull_death - asse...

The only workaround I found is to specify explicitly COLUMNS env var to be of some higher value on CI, but that would make the entire log wider, so wider section headers etc.

Ideally I wish there was some pytest setting which would take and not truncate the first line (i.e. split by \n) of that line to be displayed so in the captured logs we have it in its entirety.

nicoddemus commented 2 years ago

Perhaps it makes sense for us to detect we are running on CI and not truncate at all? I recall we already do something similar to truncate tracebacks...

yarikoptic commented 2 years ago

not sure if worth "on CI" -- the same effect is observed when stdout is not a terminal, i.e. if gets piped to | cat.

nicoddemus commented 2 years ago

Fair enough.

david1542 commented 2 years ago

Can you provide a reproducible example? I'd like to take a stab at this :)

yarikoptic commented 2 years ago 
#!/bin/bash
set -x

cd "$(mktemp -d ${TMPDIR:-/tmp}/pytest-XXXXXXX)"

cat >| test_some_path_could_be_long.py <<EOF
def test_short_fail():
    assert False 
def test_veryyyyyyyyy_long_test_name_just_because_no_long_path():
    assert False  # which could be long line too
EOF

python3 -m pytest test_some_path_could_be_long.py 
python3 -m pytest test_some_path_could_be_long.py | cat

would end with

FAILED test_some_path_could_be_long.py::test_short_fail - assert False
FAILED test_some_path_could_be_long.py::test_veryyyyyyyyy_long_test_name_just_because_no_long_path

so no the assert False like in previous output without | cat

yarikoptic commented 1 year ago

ping on this

tunstek commented 1 year ago

Could we have a flag that just stops the truncation of the summary? I would personally just use that in general, even outside of CI.