search

testng-team / testng

TestNG testing framework
https://testng.org
Apache License 2.0
1.98k stars 1.02k forks source link

Feature request: Add new `fail` config failure policy #2860

Open pavetok opened 1 year ago

pavetok commented 1 year ago

TestNG Version

LATEST

Problem

Can't to fail fast CI build right after configuration method failure.

Maven Surefire has option skipAfterFailureCount. It allows us to fail E2E tests build after first test method failure. So we can investigate issue in relevant stand state. But we can't achieve the same for configuration method failures.

Solution

Add new fail config failure policy. So all test methods in test class would be failed and CI build would be failed asap too.

krmahadevan commented 1 year ago

@pavetok - TestNG is already returning back a return code of 1 when there are configuration failures. Have you tried running a test at the command prompt and then printing the last return code using echo $? ? You should see a non negative value. Not sure what additional can be done here in terms of failure policy, when its already being done by TestNG. Maybe you may have to look at your CI system to find out what may be going wrong ?

krmahadevan commented 1 year ago

@pavetok - If you are using Jenkins, you should perhaps check and ensure that the recommendation in this stackoverflow thread is done.

Add -Dmaven.test.failure.ignore=false to MAVEN_OPTS.

pavetok commented 1 year ago

If you are using Jenkins, you should perhaps check and ensure that the recommendation in this stackoverflow thread is done.

@krmahadevan - We are using TeamCity and by default this option disabled in the surefire plugin https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#testFailureIgnore

pavetok commented 1 year ago

TestNG is already returning back a return code of 1 when there are configuration failures.

@krmahadevan - Code of 1 will be returned after process exit at the very end of test run. It seems it can't be used for the purpose of a fast fail.

krmahadevan commented 1 year ago

@krmahadevan - We are using TeamCity and by default this option disabled in the surefire plugin

If you read the link that I shared, it also talks about the same thing but this gets enabled by Jenkins. Maybe TeamCity is doing the same thing too? Have you tried setting it and then observing if there is any difference in behavior ?

@krmahadevan - Code of 1 will be returned after process exit at the very end of test run. It seems it can't be used for the purpose of a fast fail.

The exit will depend on which configuration had a failure. If it was a @BeforeSuite it would be immediate. If it was a @BeforeTest that particular <test> will be skipped, but TestNG will proceed to execute the next <test> tag and so on for @BeforeClass etc.,

If you would want to abort the execution from moving further after any failure, I think you can do the same with a TestNG listener, but throwing a SkipException