search

pytest-dev / pytest

The pytest framework makes it easy to write small tests, yet scales to support complex functional testing
https://pytest.org
MIT License
12.16k stars 2.69k forks source link

`=== short test summary info ===` messages are not useful for `ExceptionGroup`s #12943

Open Zac-HD opened 2 weeks ago

Zac-HD commented 2 weeks ago

If there is only a single leaf exception, at any depth, I suggest we produce the short summary message for that leaf exception, and append [single exception in an ExceptionGroup] to the message.

nicoddemus commented 2 weeks ago

bikeshedding: [single exception in an ExceptionGroup] to make it clear it is a single exception, and avoid someone thinking "ok and about the other exceptions in that exception group"?

jakkdl commented 6 days ago

One problem with appending the [single exc...] is it getting lost if the error message is very long and gets truncated, making it impossible to distinguish a wrapped exception from a non-wrapped exception.

FAILED foo.py::test_bar - ValueError: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...

so unless we want custom logic for the truncation in _format_trimmed/_get_line_with_reprcrash_message it's probably better to prepend any message.

(you don't get truncation if you run in CI or with -vv, but day-to-day users will encounter it)

Zac-HD commented 6 days ago

I think it's okay to truncate here; I'd rather lose the note than the actual message at least - and the group will still be visible if you go look at the specific test.

jakkdl commented 2 days ago

I think there's cases where losing the message would be preferably to losing the note, e.g. if you're writing anything equivalent to

with pytest.raises(ValueError):
    raise ExceptionGroup("", [ValueError("."*100])

then the reason it fails is because it's in a group. And if we make the note somewhat brief it's not going to take up much space, and the reader can quickly skip it when it's been read a million times.

and with a leading note we can easily get a format that allows printing multiple exceptions in a logical way:

[in ExceptionGroup]: ValueError("my message"), TypeError("my other message")

Though with -vv we might want to display the full exceptiongroup, nested and all. While getting exactly >1 sub-exceptions might be less common than exactly one, the message ExceptionGroup: bbbbbbbbbb (2 sub-exceptions) is about as useless as with one sub-exception and we could probably print both sub-exceptions without any issues.

I suppose truncation introduces another issue here, where you could lose info about the number of exceptions, but we can add that back to the note

[2 in ExceptionGroup]: ValueError("my message"), TypeError("my other message")