search

testng-team / testng

TestNG testing framework
https://testng.org
Apache License 2.0
1.99k stars 1.02k forks source link

Different instances of ITestResult with different IAttributes while failing in configuration #409

Closed outofrange closed 4 years ago

outofrange commented 11 years ago

I am using a TestListener to catch errors and store the exception message into the IAttributes of ITestResult (see https://github.com/outofrange/testresultbug/blob/master/testresultbug/src/main/java/org/outofrange/bugproject/MyListener.java)

  1. If a test fails during it's test method, everything is fine and my @AfterMethod is able to gain the correct information from the stored exception message.
  2. But if a test fails during configuration, the instances of ITestResult given to my TestListener and to my @AfterMethod are not the same; I can't access the information stored in my listener.

Debugging in Eclipse also shows different ids for ITestResult in the listener and my teardown method.

I would expect the same behaviour of TestNG in both cases, giving me the same instance of ITestResult.

Example test for case 1.: https://github.com/outofrange/testresultbug/blob/master/testresultbug/src/main/java/org/outofrange/bugproject/MyTestFailure.java Output: Detected a failure, setting failreason Test failed because of: AssertionError: failure in test method expected [true] but found [false]

Example test for case 2.: https://github.com/outofrange/testresultbug/blob/master/testresultbug/src/main/java/org/outofrange/bugproject/MyConfigurationFailure.java Output: Detected a failure, setting failreason Test failed because of: null

I've created a complete example maven project (using TestNG 6.8.5) here: https://github.com/outofrange/testresultbug

outofrange commented 10 years ago

This issue still exists with TestNG 6.8.7.

I would try to take a deeper look into it, if somebody could confirm this as a bug (and not expected behaviour).

krmahadevan commented 4 years ago

TestNG injects an ITestResult object for the @Test method that got executed, into @AfterMethod annotated method.

But for the following methods, its the same ITestResult object for a configuration method that gets executed.

  1. org.testng.IConfigurationListener#beforeConfiguration
  2. org.testng.IInvokedMethodListener#beforeInvocation(org.testng.IInvokedMethod, org.testng.ITestResult)
  3. org.testng.IInvokedMethodListener#afterInvocation(org.testng.IInvokedMethod, org.testng.ITestResult)
  4. org.testng.IConfigurationListener#onConfigurationFailure

Closing this issue (TestNG is working as designed in terms of not injecting the ITestResult object of a failed configuration in an @AfterMethod method)