search

testng-team / testng

TestNG testing framework
https://testng.org
Apache License 2.0
1.99k stars 1.02k forks source link

Failure in one test class skips all test classes in a given suite #2277

Open abudhakeer opened 4 years ago

abudhakeer commented 4 years ago

TestNG Version

6.9.5

Expected behavior

Even if config failure happens in one test class the remaining test classes from that suite should be executed.

Actual behavior

When a test class fails due to config failure the remaining test classes from that suite is skipped.

Is the issue reproductible on runner?

Test case sample

There are test classes (T1 to T5) in a given suite. All these test class depend a common test base TB. I see this behaviour even though i set the "configureFailurePolicy" to "continue".

One weird thing which i see here is, Consider the @BeforeClass method of T1 and T2 contains "Assert.fail()". During the execution of BeforeClass of T1 the failure is reported and all the test cases of T1 are skipped as expected.

When the BeforeClass of T2 is executed, the failure is reported and the test is halted. Neither the BeforeMethod of T2 is executed nor the test classes T3 to T5.

If the failure should skip test classes it should everything else it should work for all test classes. This behaviour is little weird.

I did look into other similar issues like - https://github.com/cbeust/testng/issues/739, https://github.com/cbeust/testng/issues/471 etc. I can find the fixes are present in my version of TestNG still i face this issue.

Can someone explain what could be done to solve this issue with the existing version of 6.9.5?

krmahadevan commented 4 years ago

@abudhakeer - The latest released version of TestNG is 7.1.0. Please retry using this version and if the problem still exists, please help add a sample that can be used to reproduce the problem.

abudhakeer commented 4 years ago

@krmahadevan Thanks for your prompt response. I cannot upgrade the version since we depend on an internal framework which is using 6.9.5 version of testNG. If we try to upgrade, the functionalities provided by them could be broken or stop working. I tried to look into the fixes made in above mentioned similar issues, i find those to be present in my version of TestNG.

Could you please help in figuring out the fixes made to resolve this or any solution with this version? If this is not possible then i can speak up with the framework team to do the upgradation. But i just wanna try before moving forward.

krmahadevan commented 4 years ago

@abudhakeer - Its not going to be possible to apply fixes for older versions... You would need to upgrade to the latest released version, and if the problem still persists, please include a sample that can be used to reproduce the problem.

mattrench commented 3 years ago

I am on TestNG version 7.3.0 and I am seeing the same problem. If one test fails in the beforeMethod or afterMethod function, it skips all other tests in the class. I have the same aforementioned setup with a superclass before and after method existing, but not related to the failure.