Hard fail on layer set up

zopefoundation / zope.testrunner

This package provides a flexible test runner with layer support.

https://zopetestrunner.readthedocs.io/

Other

2 stars 23 forks source link

Hard fail on layer set up #175

Open Andmedoctopus opened 4 months ago

Andmedoctopus commented 4 months ago

If layer failed to setup testrunner still continue to run tests which might have side effect on other testes. Layer setup should fail hard if error happened.

--stop-on-error is not an option, because it also stop any failed tests which is unacceptable for CI

d-maurer commented 4 months ago

Andrei Medvedev wrote at 2024-6-26 23:40 -0700:

If layer failed to setup testrunner still continue to run tests which might have side effect on other testes. Layer setup should fail hard if error happened.

--stop-on-layer-error is not an option, because it also stop any failed tests which is unacceptable for CI

Could you explain what "stop any failed tests" should mean? As the tests (apparently) have failed, they can not be stopped (but the corresponding failure output might be lost).

Andmedoctopus commented 4 months ago

Andrei Medvedev wrote at 2024-6-26 23:40 -0700: If layer failed to setup testrunner still continue to run tests which might have side effect on other testes. Layer setup should fail hard if error happened. --stop-on-layer-error is not an option, because it also stop any failed tests which is unacceptable for CI Could you explain what "stop any failed tests" should mean? As the tests (apparently) have failed, they can not be stopped (but the corresponding failure output might be lost).

I'm sorry. I meant flag --stop-on-error. This flag will stop either layer setup or first failed test. But there is no option to fail only on layer setup.

My point is if layer fail to setup there's no reason to try to run a test which require failed layer

Andmedoctopus commented 4 months ago

@d-maurer could you take a look, please?

d-maurer commented 4 months ago

Andrei Medvedev wrote at 2024-7-1 01:45 -0700:

@d-maurer could you take a look, please?

I already have left a comment:

I still do not yet know what exactly your change should do.

Likely, it should skip all tests belonging to a layer when the layer setup failed -- but is this not already the case?
We would like to have a test. The test would in addition clarify when your change gets effective and what effect the PR plans to achieve.
The change should be decribed in CHANGES.rst to document what has been changed for what purpose and what is the new behavior.

icemac commented 3 months ago

@Andmedoctopus Are you still working on this PR otherwise I'd close it as stalled.

Andmedoctopus commented 3 months ago

@icemac I'm working on another task now and planning to return to this PR just after that. Please don't close it. If needed I can mark it as draft for easier management