search

testng-team / testng

TestNG testing framework
https://testng.org
Apache License 2.0
1.98k stars 1.02k forks source link

Missing "before suite" callback when running JUnit test? #2556

Open vlsi opened 3 years ago

vlsi commented 3 years ago

There's an AssertionFailure coming from Gradle.

Gradle code: https://github.com/gradle/gradle/blob/06140ac5fbdce282f868197064156b9b1753012f/subprojects/testing-jvm/src/main/java/org/gradle/api/internal/tasks/testing/testng/TestNGTestResultProcessorAdapter.java#L184

// Gradle code:

            parentId = testMethodParentId.get(iTestResult.getMethod());
            assert parentId != null;

I guess that they expect that each method should have a parent (suite?)

I do not completely follow the logic here, however, my understanding is: a) It could be TestNG defect, and TestNG should call onStart(ITestContext iTestContext) or onBeforeClass(ITestClass testClass) before running JUnit methods b) It might be a false assumption by Gradle. In that case, they need to adjust the code.

TestNG Running Junit Tests > Junit via TestNG XML STANDARD_ERROR
    [TestRunner] [ERROR] 
    Failure in JUnit mode for class test.junit.SetNameTest
    org.testng.TestNGException: 
    Failure in JUnit mode for class test.junit.SetNameTest
        at org.testng.junit.JUnit4TestRunner.start(JUnit4TestRunner.java:107)
        at org.testng.junit.JUnit4TestRunner.run(JUnit4TestRunner.java:58)
        at org.testng.TestRunner$1.run(TestRunner.java:672)
        at java.util.ArrayList.forEach(ArrayList.java:1249)
        at org.testng.TestRunner.runJUnitWorkers(TestRunner.java:871)
        at org.testng.TestRunner.privateRunJUnit(TestRunner.java:701)
        at org.testng.TestRunner.run(TestRunner.java:594)
        at org.testng.SuiteRunner.runTest(SuiteRunner.java:421)
        at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:415)
        at org.testng.SuiteRunner.privateRun(SuiteRunner.java:376)
        at org.testng.SuiteRunner.run(SuiteRunner.java:320)
        at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
        at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
        at org.testng.TestNG.runSuitesSequentially(TestNG.java:1220)
        at org.testng.TestNG.runSuitesSequentially(TestNG.java:1215)
        at org.testng.TestNG.runSuitesLocally(TestNG.java:1142)
        at org.testng.TestNG.runSuites(TestNG.java:1071)
        at org.testng.TestNG.run(TestNG.java:1039)
        at org.gradle.api.internal.tasks.testing.testng.TestNGTestClassProcessor.runTests(TestNGTestClassProcessor.java:141)
        at org.gradle.api.internal.tasks.testing.testng.TestNGTestClassProcessor.stop(TestNGTestClassProcessor.java:90)
        at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:61)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
        at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
        at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
        at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
        at com.sun.proxy.$Proxy2.stop(Unknown Source)
        at org.gradle.api.internal.tasks.testing.worker.TestWorker.stop(TestWorker.java:133)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
        at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
        at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
        at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
        at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
        at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
        at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
        at java.lang.Thread.run(Thread.java:748)
    Caused by: java.lang.AssertionError
        at org.gradle.api.internal.tasks.testing.testng.TestNGTestResultProcessorAdapter.onTestStart(TestNGTestResultProcessorAdapter.java:185)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:104)
        at org.gradle.api.internal.tasks.testing.testng.TestNGListenerAdapterFactory$AdaptedListener.invoke(TestNGListenerAdapterFactory.java:101)
        at org.gradle.api.internal.tasks.testing.testng.TestNGListenerAdapterFactory$AdaptedListener.invoke(TestNGListenerAdapterFactory.java:95)
        at com.sun.proxy.$Proxy4.onTestStart(Unknown Source)
        at org.testng.junit.JUnit4TestRunner$RL.testStarted(JUnit4TestRunner.java:210)
        at org.junit.runner.notification.SynchronizedRunListener.testStarted(SynchronizedRunListener.java:80)
        at org.junit.runner.notification.RunNotifier$5.notifyListener(RunNotifier.java:156)
        at org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72)
        at org.junit.runner.notification.RunNotifier.fireTestStarted(RunNotifier.java:153)
        at org.junit.internal.runners.JUnit38ClassRunner$OldTestClassAdaptingListener.startTest(JUnit38ClassRunner.java:41)
        at junit.framework.TestResult.startTest(TestResult.java:168)
        at junit.framework.TestResult.run(TestResult.java:119)
        at junit.framework.TestCase.run(TestCase.java:130)
        at junit.framework.TestSuite.runTest(TestSuite.java:241)
        at junit.framework.TestSuite.run(TestSuite.java:236)
        at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:90)
        at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
        at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
        at org.testng.junit.JUnit4TestRunner.start(JUnit4TestRunner.java:73)
        ... 45 more
juherr commented 3 years ago

That's strange we've never seen the issue before. I don't know yet what is the TestNG expected behavior.

Could you provide a sample that will allow us to reproduce the issue easily?

vlsi commented 3 years ago

It comes from the already existing test:

<suite name="TestNG Running Junit Tests">

  <test name="Junit via TestNG XML" junit="true">
    <classes>
      <class name="test.junit.SetNameTest" />
    </classes>
  </test>

</suite>

However, for some reason, Gradle does not recognize the failure and it treats the outcome as "zero tests found", so you do not see the build failure:

WARNING   0.1sec,    0 completed,   0 failed,   0 skipped, Junit via TestNG XML
juherr commented 3 years ago

Ok, thanks! Did you open an issue for Gradle? I think its behavior is not the expected one because it provides some false-positive and we can miss some test issues :(