search

testng-team / testng

TestNG testing framework
https://testng.org
Apache License 2.0
1.99k stars 1.02k forks source link

TestNG fails to finish gracefully when executing tests with the dependsOnMethods attribute if an exception happens in ITestListener.onTestStart #2960

Open harmin-parra opened 1 year ago

harmin-parra commented 1 year ago

TestNG Version

7.8.0

Description

If I have a test annotation with the dependsOnMethods attribute, and if an exception occurs in ITestListener.onTestStart, TestNG is unable to finish the test execution and exists in an error state.

ITestListener.onTestSkipped never gets called for the test with the dependsOnMethods attribute.

The test result summary never gets displayed in the console.

Expected behavior

TestNG executes and displays the test result summary in the console. 'onTestSkipped' gets printed twice.

   [testng] onTestStart
   [testng] onTestSkipped
   [testng] Aug 16, 2023 1:33:46 PM org.testng.log4testng.Logger info
   [testng] INFO: [TestNG] Running:
   [testng]   /home/harmin/.eclipse-workspace2/testng-bug/testng.xml
   [testng] onTestStart
   [testng] onTestSkipped
   [testng] ===============================================
   [testng] All Tests
   [testng] Total tests run: 2, Passes: 0, Failures: 0, Skips: 2
   [testng] ===============================================

Actual behavior

TestNG doesn't execute until the end. The test result summary never gets displayed. "onTestSkipped" gets printed once.

   [testng] Aug 16, 2023 1:34:15 PM org.testng.log4testng.Logger info
   [testng] INFO: [TestNG] Running:
   [testng]   /home/harmin/.eclipse-workspace2/testng-bug/testng.xml
   [testng] onTestStart
   [testng] onTestSkipped
   [testng] onTestStart
   [testng] Exception in thread "main" org.testng.SkipException: Skip
   [testng]     at TestListener.onTestStart(TestListener.java:14)
   [testng]     at org.testng.internal.TestListenerHelper.runTestListeners(TestListenerHelper.java:115)
   [testng]     at org.testng.internal.invokers.TestInvoker.runTestResultListener(TestInvoker.java:262)
   [testng]     at org.testng.internal.invokers.TestInvoker.registerSkippedTestResult(TestInvoker.java:823)
   [testng]     at org.testng.internal.invokers.ITestInvoker.registerSkippedTestResult(ITestInvoker.java:41)
   [testng]     at org.testng.internal.invokers.TestInvoker.invokeTestMethods(TestInvoker.java:155)
   [testng]     at org.testng.internal.invokers.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:148)
   [testng]     at org.testng.internal.invokers.TestMethodWorker.run(TestMethodWorker.java:128)
   [testng]     at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
   [testng]     at org.testng.TestRunner.privateRun(TestRunner.java:848)
   [testng]     at org.testng.TestRunner.run(TestRunner.java:621)
   [testng]     at org.testng.SuiteRunner.runTest(SuiteRunner.java:443)
   [testng]     at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:437)
   [testng]     at org.testng.SuiteRunner.privateRun(SuiteRunner.java:397)
   [testng]     at org.testng.SuiteRunner.run(SuiteRunner.java:336)
   [testng]     at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
   [testng]     at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:95)
   [testng]     at org.testng.TestNG.runSuitesSequentially(TestNG.java:1280)
   [testng]     at org.testng.TestNG.runSuitesLocally(TestNG.java:1200)
   [testng]     at org.testng.TestNG.runSuites(TestNG.java:1114)
   [testng]     at org.testng.TestNG.run(TestNG.java:1082)
   [testng]     at org.testng.TestNG.privateMain(TestNG.java:1440)
   [testng]     at org.testng.TestNG.main(TestNG.java:1404)
   [testng] The tests failed.

Is the issue reproducible on runner?

Test case sample

See repo with sample code https://github.com/harmin-parra/issue-2960

Info

If I replace @Test(dependsOnMethods = {"test1"}) by @Test, TesnNG executes until the end and I get the test result summary

   [testng] ===============================================
   [testng] All Tests
   [testng] Total tests run: 2, Passes: 0, Failures: 0, Skips: 2
   [testng] ===============================================
juherr commented 1 year ago

There is no expected exception in listener interfaces: you are supposed to manage exceptions in the listener implementation.

But TestNG can do better and the expected behavior from listeners should be the same as through annotated methods.

Could you check and tell us what is happening when a SkipException is thrown from a @BeforeTest method?

harmin-parra commented 1 year ago

The exception seems to be well handled in the @BeforeTest method.

I observe the same behavior and outcome.

No test result summary at the end of the execution.

   [testng] Aug 16, 2023 6:38:37 PM org.testng.log4testng.Logger info
   [testng] INFO: [TestNG] Running:
   [testng]   /home/harmin/.eclipse-workspace2/testng-bug/testng.xml
   [testng] onTestStart
   [testng] onTestFailure
   [testng] onTestStart
   [testng] Exception in thread "main" org.testng.SkipException: Skip
   [testng]     at TestListener.onTestStart(TestListener.java:14)
   [testng]     at org.testng.internal.TestListenerHelper.runTestListeners(TestListenerHelper.java:115)
   [testng]     at org.testng.internal.invokers.TestInvoker.runTestResultListener(TestInvoker.java:262)
   [testng]     at org.testng.internal.invokers.TestInvoker.registerSkippedTestResult(TestInvoker.java:823)
   [testng]     at org.testng.internal.invokers.ITestInvoker.registerSkippedTestResult(ITestInvoker.java:41)
   [testng]     at org.testng.internal.invokers.TestInvoker.invokeTestMethods(TestInvoker.java:155)
   [testng]     at org.testng.internal.invokers.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:148)
   [testng]     at org.testng.internal.invokers.TestMethodWorker.run(TestMethodWorker.java:128)
   [testng]     at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
   [testng]     at org.testng.TestRunner.privateRun(TestRunner.java:848)
   [testng]     at org.testng.TestRunner.run(TestRunner.java:621)
   [testng]     at org.testng.SuiteRunner.runTest(SuiteRunner.java:443)
   [testng]     at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:437)
   [testng]     at org.testng.SuiteRunner.privateRun(SuiteRunner.java:397)
   [testng]     at org.testng.SuiteRunner.run(SuiteRunner.java:336)
   [testng]     at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
   [testng]     at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:95)
   [testng]     at org.testng.TestNG.runSuitesSequentially(TestNG.java:1280)
   [testng]     at org.testng.TestNG.runSuitesLocally(TestNG.java:1200)
   [testng]     at org.testng.TestNG.runSuites(TestNG.java:1114)
   [testng]     at org.testng.TestNG.run(TestNG.java:1082)
   [testng]     at org.testng.TestNG.privateMain(TestNG.java:1440)
   [testng]     at org.testng.TestNG.main(TestNG.java:1404)
   [testng] The tests failed.
krmahadevan commented 1 year ago

@harmin-parra - Listeners and configuration methods are IMO two different things. While a listener can also be made to work as if it were a configuration method, I dont think that listeners should be perceived that way.

Listeners are essentially supposed to work as if they were call backs which lets you run some additional code before/after something happens.

Configuration methods are supposed to be a subset of these callback functionalities that listener provides and are strictly confined to the lifecycle of tests.

I say this because, currently we can have a listener to report test reports, but there's no configuration method similar to that.

We can have a listener that can be configured to alter a suite (via IAlterSuiteListener) (or) be invoked right before execution starts and before execution ends (via IExecutionListener).

So not sure if we can look at both of them that way. That perhaps is why TestNG does not handle any exceptions that can arise out of the listeners.

@juherr - What do you suggest should be the further course of action on this? I feel that we just need to make it explicit in documentation that listeners are not supposed to throw exceptions and they should be handling them on their own.

juherr commented 1 year ago

Yes. The documentation should be updated.

But I don't see good reason that explains the failure here.

krmahadevan commented 1 year ago

@juherr - The listener is explicitly throwing an exception

See here https://github.com/harmin-parra/issue-2960/blob/master/src/TestListener.java#L14

That perhaps explains why the failure occurs

Can be easily simulated using the below sample

import org.testng.ITestListener;
import org.testng.ITestResult;
import org.testng.SkipException;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

@Listeners(SampleTestClass.TestListener.class)
public class SampleTestClass {

    @Test
    public void testMethod() {
    }

    @Test(dependsOnMethods = "testMethod")
    public void anotherTestMethod() {
    }

    public static class TestListener implements ITestListener {
        @Override
        public void onTestStart(ITestResult result) {
            throw new SkipException("Don't run");
        }
    }
}
krmahadevan commented 1 year ago

Execution logs look like below

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.1.2:test (default-test) on project testng_playground: 
[ERROR] 
[ERROR] Please refer to /Users/kmahadevan/githome/playground/testng_playground/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] There was an error in the forked process
[ERROR] Don't run
[ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: There was an error in the forked process
[ERROR] Don't run
[ERROR]         at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:628)
[ERROR]         at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:285)
[ERROR]         at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:250)
[ERROR]         at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1203)
[ERROR]         at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1055)
[ERROR]         at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:871)
[ERROR]         at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:126)
[ERROR]         at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2(MojoExecutor.java:328)
[ERROR]         at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute(MojoExecutor.java:316)
[ERROR]         at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
[ERROR]         at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:174)
[ERROR]         at org.apache.maven.lifecycle.internal.MojoExecutor.access$000(MojoExecutor.java:75)
[ERROR]         at org.apache.maven.lifecycle.internal.MojoExecutor$1.run(MojoExecutor.java:162)
[ERROR]         at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute(DefaultMojosExecutionStrategy.java:39)
[ERROR]         at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:159)
[ERROR]         at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:105)
[ERROR]         at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:73)
[ERROR]         at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:53)
[ERROR]         at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:118)
[ERROR]         at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:261)
[ERROR]         at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:173)
[ERROR]         at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:101)
[ERROR]         at org.apache.maven.cli.MavenCli.execute(MavenCli.java:906)
[ERROR]         at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:283)
[ERROR]         at org.apache.maven.cli.MavenCli.main(MavenCli.java:206)
[ERROR]         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[ERROR]         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[ERROR]         at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[ERROR]         at java.base/java.lang.reflect.Method.invoke(Method.java:566)
[ERROR]         at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:283)
[ERROR]         at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:226)
[ERROR]         at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:407)
[ERROR]         at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:348)
[ERROR] 
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
➜  testng_playground
juherr commented 1 year ago

Sure but I remember we catched exceptions from listener with a warning. Why not here?

krmahadevan commented 1 year ago

Do you remember for which listener we were catching exceptions?

juherr commented 1 year ago

Not sure but I think it was before/afterMethod

harmin-parra commented 1 year ago

The reason I am throwing an exception in a Listener is because I want to be able to skip the remaining tests of a suite if a test fails.

And I don't want to use the dependsOnMethods annotation in every single method, all over the code. The overhead of maintaining such annotation is just too expensive if I add / remove / rename test methods to a suite.

I guess this issue could be rejected if you agree to an evolution and decide provide a command-line parameter option called --testfailurepolicy that would take skip|continue as value and will work similar to the already existing --configfailurepolicy.

krmahadevan commented 1 year ago

@harmin-parra - That's a good suggestion. The only challenge with that would be to determine the failure strategy when @Test methods are run in parallel. In that case, what do we fail ?

On a side note, what you are asking for, can be accomplished with TestNG using 7.8.0 without needing any additional changes.

Test classes would look like below

package com.rationaleemotions;

import org.testng.Assert;
import org.testng.annotations.Test;

public class FirstClassTest {

    @Test
    public void aTest1() {}

    @Test
    public void failingTest() {
        Assert.fail("I am to be failing");
    }
}
package com.rationaleemotions;

import org.testng.Assert;
import org.testng.annotations.Test;

public class SecondClassTest {

    @Test
    public void skippingTest1() {
        Assert.fail("I should NOT have been executed");
    }

    @Test
    public void skippingTest2() {
        Assert.fail("I should NOT have been executed");
    }
}

The sample listener which manages the fail fast ask

package com.rationaleemotions;

import org.testng.*;
import org.testng.xml.XmlSuite;

import java.util.List;
import java.util.Set;

public class FailFastListener implements ITestListener, IReporter {

    @Override
    public void onTestStart(ITestResult result) {
        if (result.getTestContext().getFailedTests().size() > 0) {
            result.setStatus(ITestResult.SKIP);
            result.setThrowable(new SkipException("Skipping this test"));
        }
    }

    @Override
    public void generateReport(List<XmlSuite> xmlSuites, List<ISuite> suites, String outputDirectory) {
        suites.stream()
                .flatMap(each -> each.getResults().values().stream())
                .map(ISuiteResult::getTestContext)
                .forEach(each -> {
                    print("Passed Tests", each.getPassedTests().getAllResults());
                    print("Failed Tests", each.getFailedTests().getAllResults());
                    print("Skipped Tests", each.getSkippedTests().getAllResults());
                });
    }

    private static void print(String msg, Set<ITestResult> results) {
        System.err.println(msg);
        results.forEach(each -> System.err.println(each.getMethod().getQualifiedName()));
    }
}

Here's a suite file that uses all of this

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Practice Suite" verbose="2">
    <listeners>
        <listener class-name="com.rationaleemotions.FailFastListener"/>
    </listeners>
    <test name="Test Basics 1" verbose="2" parallel="false">
        <classes>
            <class name="com.rationaleemotions.FirstClassTest"/>
            <class name="com.rationaleemotions.SecondClassTest"/>
        </classes>
    </test> <!-- Test -->
</suite> <!-- Suite -->

Here's the execution output

===============================================
    Test Basics 1
    Tests run: 4, Failures: 1, Skips: 2
===============================================

Passed Tests
com.rationaleemotions.FirstClassTest.aTest1
Failed Tests
com.rationaleemotions.FirstClassTest.failingTest
Skipped Tests
com.rationaleemotions.SecondClassTest.skippingTest2
com.rationaleemotions.SecondClassTest.skippingTest1
[ERROR] Tests run: 4, Failures: 1, Errors: 0, Skipped: 2, Time elapsed: 0.487 s <<< FAILURE! -- in TestSuite
[ERROR] com.rationaleemotions.FirstClassTest.failingTest -- Time elapsed: 0.003 s <<< FAILURE!
java.lang.AssertionError: I am to be failing
        at org.testng.Assert.fail(Assert.java:111)
        at com.rationaleemotions.FirstClassTest.failingTest(FirstClassTest.java:14)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
        at org.testng.internal.invokers.TestInvoker.invokeMethod(TestInvoker.java:664)
        at org.testng.internal.invokers.TestInvoker.invokeTestMethod(TestInvoker.java:227)
        at org.testng.internal.invokers.MethodRunner.runInSequence(MethodRunner.java:50)
        at org.testng.internal.invokers.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:957)
        at org.testng.internal.invokers.TestInvoker.invokeTestMethods(TestInvoker.java:200)
        at org.testng.internal.invokers.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:148)
        at org.testng.internal.invokers.TestMethodWorker.run(TestMethodWorker.java:128)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
        at org.testng.TestRunner.privateRun(TestRunner.java:848)
        at org.testng.TestRunner.run(TestRunner.java:621)
        at org.testng.SuiteRunner.runTest(SuiteRunner.java:443)
        at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:437)
        at org.testng.SuiteRunner.privateRun(SuiteRunner.java:397)
        at org.testng.SuiteRunner.run(SuiteRunner.java:336)
        at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
        at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:95)
        at org.testng.TestNG.runSuitesSequentially(TestNG.java:1280)
        at org.testng.TestNG.runSuitesLocally(TestNG.java:1200)
        at org.testng.TestNG.runSuites(TestNG.java:1114)
        at org.testng.TestNG.run(TestNG.java:1082)
        at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:308)
        at org.apache.maven.surefire.testng.TestNGXmlTestSuite.execute(TestNGXmlTestSuite.java:71)
        at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:113)
        at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:385)
        at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
        at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:507)
        at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:495)

[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   FirstClassTest.failingTest:14 I am to be failing
[INFO] 
[ERROR] Tests run: 4, Failures: 1, Errors: 0, Skipped: 2
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  1.982 s
[INFO] Finished at: 2023-09-10T19:41:04+05:30