search

pytest-dev / pytest-timeout

MIT License
213 stars 64 forks source link

Are timeouts supposed to result in Error or Failure or both? #171

Open kaddkaka opened 7 months ago

kaddkaka commented 7 months ago

I'm running a smaller testsuite and moste of the tests fail 10 seconds timeout because of slow setup code (currently residing inside the tests). Some of the timeouted tests result in Error some in Failure:

FAILED base/software/test_external_call.py::TestExtCall::test_can_call_external_lib[/local/path.slow-mode] - Failed: Timeout >10.0s
ERROR base/software/test_external_call.py::TestExtCall::test_can_call_external_lib[/local/path.fast-mode] - Failed: Timeout >10.0s

Why? Is this intentional?

If I increase my timeout the ERRORs seem to go away and all timeouted tests are listed as FAILUREs.

RonnyPfannschmidt commented 7 months ago

pytest considers exception/errors in setup/teardown an error (as that shouldnt happen) if it happens in a test, its as expected

i beleive timeout should transition to a new outcome type as a timeout is more similar to a xfail

kaddkaka commented 7 months ago

pytest considers exception/errors in setup/teardown an error (as that shouldnt happen) if it happens in a test, its as expected

i beleive timeout should transition to a new outcome type as a timeout is more similar to a xfail

That sounds reasonable and "Timeout" outcome sounds good. Is there work ongoing or another issue to link this one to?

RonnyPfannschmidt commented 7 months ago

Not yet

flub commented 7 months ago

On Fri 22 Mar 2024 at 02:48 -0700, Ronny Pfannschmidt wrote:

i beleive timeout should transition to a new outcome type as a timeout is more similar to a xfail

can plugins define new outcome types?

The-Compiler commented 7 months ago

I believe plugins can set any outcome as long as they handle it appropriately in a pytest_report_teststatus.