Issues with new requirement of testng for testing

[rsearls]

Quarkus 1.2.0 now requires testng. This has introduced the following 2 issues. There are 2 issues.

• When 2 or more tests are run, testng prints an error in usage message.
• The names of the individual tests that successfully run are no longer printed. Only a summary of total tests run is listed.

Expected behavior

• Each test file that successfully runs reports the filename and test summary. At the end of the test suite a summary of the testsuite is printed.
• No error msgs from testng

Actual behavior

• Only the final testsuite summary is printed. None of the names of the test files are listed.

• The following msg is printed when 2 or more tests are in the testsuite.

  [INFO] Running TestSuite

  [TestNG] [ERROR] No test suite found. Nothing to run

  Usage:

  [options] The XML suite files to run

  Options: -configfailurepolicy Configuration failure policy (skip or continue) -d Output directory -dataproviderthreadcount Number of threads to use when running data providers -excludegroups Comma-separated list of group names to exclude -groups Comma-separated list of group names to be run -junit JUnit mode Default: false -listener List of .class files or list of class names implementing ITestListener or ISuiteListener -methods Comma separated of test methods Default: [] -methodselectors List of .class files or list of class names implementing IMethodSelector -mixed Mixed mode - autodetect the type of current test and run it with appropriate runner Default: false -objectfactory List of .class files or list of class names implementing ITestRunnerFactory -parallel Parallel mode (methods, tests or classes) Possible Values: [tests, methods, classes, instances, none, true, false] -port The port -reporter Extended configuration for custom report listener -suitename Default name of test suite, if not specified in suite definition file or source code -suitethreadpoolsize Size of the thread pool to use to run suites Default: 1 -testclass The list of test classes -testjar A jar file containing the tests -testname Default name of test, if not specified in suitedefinition file or source code -testnames The list of test names to run -testrunfactory, -testRunFactory The factory used to create tests -threadcount Number of threads to use when running tests in parallel -usedefaultlisteners Whether to use the default listeners Default: true -log, -verbose Level of verbosity -xmlpathinjar The full path to the xml file inside the jar file (only valid if -testjar was specified) Default: testng.xml

To Reproduce This project reproduces the issue, https://github.com/rsearls/quarkus-testng Follow the directions in the README file to setup the test properly.

Environment (please complete the following information):

• Linux localhost.localdomain 5.0.16-100.fc28.x86_64 #1 SMP Tue May 14 18:22:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
• jdk 11.0.2
• mvn 3.6.0
• quarkus 1.2.0

[gsmet]

Quarkus 1.2.0 now requires testng.

Can you expand on that? Because we are not supposed to use TestNG in Quarkus.

[rsearls]

The pom.xml provided in the sample project contains the minimum archives required to compile and run the tests. Each one of the archives was added because of a compiler or test run issue in which a needed class was not present.

Comment out the archive for testng lines 65-69. Running the test "mvn test" displays the following msg. [ERROR] org.jboss.resteasy.test.core.basic.ByeTest Time elapsed: 1.082 s <<< ERROR! java.lang.reflect.InvocationTargetException Caused by: java.lang.NoClassDefFoundError: org/testng/annotations/BeforeClass

To resolve this I had to add the testng archive.
My test code has not changed between quarkus 1.1.1.Final and 1.2.0.Final Updating to version 1.2.0.Final this issue occurred.

[rsearls]

And none of my tests classes use or import any org/testng/* classes

[rsearls]

Here is the stacktrace. The failure appears to be comming from at io.quarkus.arquillian.QuarkusTestNgCallbacks.invokeTestNgBeforeClasses(QuarkusTestNgCal lbacks.java:31)

"main@1" prio=5 tid=0x1 nid=NA runnable java.lang.Thread.State: RUNNABLE at java.lang.NoClassDefFoundError.(NoClassDefFoundError.java:59) at io.quarkus.arquillian.QuarkusTestNgCallbacks.invokeTestNgBeforeClasses(QuarkusTestNgCallbacks.java:31) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:566) at io.quarkus.arquillian.QuarkusBeforeAfterLifecycle.invokeCallbacks(QuarkusBeforeAfterLifecycle.java:83) at io.quarkus.arquillian.QuarkusBeforeAfterLifecycle.beforeClass(QuarkusBeforeAfterLifecycle.java:45) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:566) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:86) at org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:90) at org.jboss.arquillian.test.impl.TestContextHandler.createClassContext(TestContextHandler.java:83) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:566) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:86) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:95) at org.jboss.arquillian.test.impl.TestContextHandler.createSuiteContext(TestContextHandler.java:69) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:566) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:86) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:95) at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:133) at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:105) at org.jboss.arquillian.test.impl.EventTestRunnerAdaptor.beforeClass(EventTestRunnerAdaptor.java:89) at org.jboss.arquillian.junit.Arquillian$2.evaluate(Arquillian.java:163) at org.jboss.arquillian.junit.Arquillian.multiExecute(Arquillian.java:350) at org.jboss.arquillian.junit.Arquillian.access$200(Arquillian.java:54) at org.jboss.arquillian.junit.Arquillian$3.evaluate(Arquillian.java:177) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.jboss.arquillian.junit.Arquillian.run(Arquillian.java:115) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)

[kenfinnigan]

The Arquillian adapter supports JUnit and TestNG, as the MicroProfile TCKs use one or the other depending on which spec it is.

I wonder if the issue is because the reproducer has the Arquillian TestNG container as a dependency instead of the JUnit one, which is triggering the TestNG methods in the Quarkus Arquillian adapter to be called?

[stuartwdouglas]

The Aquillian adaptor supports TestNG but it is an optional dependency, and is invoked via reflection. You can only get this error if you include the TestNG arquillian support, but exclude TestNG itself. If you are not using TestNG make sure you don't have the testng arquillian support present.