search

opengeospatial / teamengine

TEAM Engine (Test, Evaluation, And Measurement Engine) is an engine for testing web services and other resources written in JAVA. It executes test scripts written in Compliance Test Language (CTL), TestNG and other languages. It is lightweight and easy to run as a command line or to setup as a service. It can be used to test any type of service or encoding. It is also the official tool used by the Open Geospatial Consortium (OGC) for compliance testing.
Apache License 2.0
45 stars 41 forks source link

Execute test with TestNG based test suite and TEAM Engine 6.0-SNAPSHOT #606

Closed dstenger closed 5 months ago

dstenger commented 7 months ago

CTL based test suites are already tested with this branch: https://github.com/opengeospatial/ets-wms13/tree/java17Tomcat101TeamEngine6

A similar branch shall be created for a TestNG based test suite.

Afterwards, some testing of all important functionalities shall be executed:

As the TestNG version was upgraded to 7.9.0 in TEAM Engine (see #599; please use the branch of that issue for testing), it is most likely that this dependency must also be updated in the test suite. Further upgrades might be necessary as the Java version was migrated to Java 17 in TEAM Engine. In addition, many dependency versions were upgraded in TEAM Engine 6.0-SNAPSHOT.

In a first step, compilation of the test suite must not necessarily be moved to Java 17 and can still be executed with Java 8 (testing with CTL based test suite was done this way). However, due to the TestNG upgrade, it is most likely that Java 8 cannot be used anymore.

dstenger commented 7 months ago

@bpross-52n Can you please following test suite for testing?

bpross-52n commented 6 months ago

Branch for tests with ets-ogcapi-processes: https://github.com/opengeospatial/ets-ogcapi-processes10/tree/java17Tomcat101TeamEngine6

bpross-52n commented 6 months ago

Branch for WFS 2.0 test suite: https://github.com/opengeospatial/ets-wfs20/tree/java17Tomcat101TeamEngine6

dstenger commented 6 months ago

Test of migration of WFS 2.0 test suite

First, I had to update the Dockerfile: https://github.com/opengeospatial/ets-wfs20/commit/02044d0fb2b13f9366caafa534c32a7f8dcf01fe

When testing the Web Browser Interface with https://cite.deegree.org/deegree-webservices-3.5.0/services/wfs200?service=WFS&request=GetCapabilities, following error occurred (failure of Main test):

Error in call to extension function {public java.lang.Object com.occamlab.te.TECore.callFunction(net.sf.saxon.expr.XPathContext,java.lang.String,java.lang.String,net.sf.saxon.om.NodeInfo) throws java.lang.Exception}: Exception in extension function net.sf.saxon.s9api.SaxonApiException: Error in call to extension function {public javax.xml.transform.Source org.opengis.cite.iso19142.TestNGController.doTestRun(org.w3c.dom.Document) throws java.lang.Exception}: Exception in extension function java.lang.ExceptionInInitializerError

Because of that, no report is generated.

Also, when trying to access the REST Interface site http://localhost:8081/teamengine/rest/suites/, the test suite is printed as ${project.name}. So, the REST Interface is also not working.

Further, the *aio.jar CLI is not working correctly (Java 17 is used for execution):

stenger@elsbeere:~/git/ets-wfs20/target$ java --version
openjdk 17.0.9 2023-10-17
OpenJDK Runtime Environment (build 17.0.9+9-Ubuntu-122.04)
OpenJDK 64-Bit Server VM (build 17.0.9+9-Ubuntu-122.04, mixed mode, sharing)
stenger@elsbeere:~/git/ets-wfs20/target$ java -jar ets-wfs20-1.41-SNAPSHOT-aio.jar ../src/main/resources/test-run-props.xml
Jan. 18, 2024 9:36:32 PM org.opengis.cite.iso19142.util.TestSuiteLogger log
INFORMATION: Using outputDirPath: /home/stenger
Exception in thread "main" java.lang.NoSuchMethodError: 'void org.testng.TestNG.addListener(org.testng.ITestNGListener)'
    at com.occamlab.te.spi.executors.testng.TestNGExecutor.execute(TestNGExecutor.java:166)
    at org.opengis.cite.iso19142.TestNGController.doTestRun(TestNGController.java:142)
    at org.opengis.cite.iso19142.TestNGController.main(TestNGController.java:82)

In addition, there are a couple of issues which must be solved when the actual migration of ets-wfs20 is done:

dstenger commented 6 months ago

Test of migration of OGC API - Processes 1.0 test suite

Used branch for testing: java17Tomcat101TeamEngine6-testng790

When building and trying to run the test suite with Docker by using command mvn clean install -Pdocker, I get the following error:

[INFO] --- docker:0.36.1:build (build) @ ets-ogcapi-processes10 ---
[ERROR] DOCKER> Cannot create docker access object  [Connect to localhost:2375 [localhost/127.0.0.1] failed: Verbindungsaufbau abgelehnt]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------

@bpross-52n Did it work on your machine or did you get a similar error?

dstenger commented 6 months ago

Problem was solved by removing dockerHost from configuration of docker-maven-plugin. Also, the version of the plugin was updated to 0.43.4.

Commit: https://github.com/opengeospatial/ets-ogcapi-processes10/commit/65934476ed61adf6df56565a98d82bf2581c7f4f

dstenger commented 6 months ago

Execute tests with ets-ogcapi-processes10

Web Browser Interface All smoke tests with Web Browser Interface were successful. It was possible to validate a server, view the different types of reports, to download and to delete the report.

REST Interface Same problem as reported in https://github.com/opengeospatial/teamengine/issues/606#issuecomment-1899176792 exists.

CLI with aio.jar It was possible to execute the CLI:

java -jar ets-ogcapi-processes10-1.2-SNAPSHOT-aio.jar ../src/main/config/test-run-props.xml
...
Test results: file:/home/stenger/testng/60e7d83e-f8fb-476d-81f2-47c31a9a0e94/testng-results.xml

Viewed reports. They looked as expected.

dstenger commented 6 months ago

Analyse problem of REST Interface

Filtering of resources was missing which was previously enabled by removed ets-common. This functionality was added: https://github.com/opengeospatial/ets-ogcapi-processes10/commit/0f585f3559d03061b17eb85ad38cac13d65d41bb

Now, the overview page of the REST Interface can be accessed: http://localhost:8081/teamengine/rest/suites/ogcapi-processes-1.0

Also, a test run can be started: http://localhost:8081/teamengine/rest/suites/ogcapi-processes-1.0/run?iut=http://tb17.geolabs.fr:8120/ogc-api/ However, there are two findings to be discussed:

Finding 1: TEAM Engine XML report is returned instead of EARL report which should be the default

<testng-results ignored="0" total="54" passed="0" failed="0" skipped="54">
  <reporter-output>
  </reporter-output>
  <suite started-at="2024-01-29T15:45:57 UTC" name="ogcapi-processes-1.0-1.2-SNAPSHOT" finished-at="2024-01-29T15:45:57 UTC" duration-ms="15">
...

Finding 2: All tests are skipped

dstenger commented 6 months ago

An issue for problem with skipped tests when using REST Interface was created in ets-ogcapi-processes10 tracker: https://github.com/opengeospatial/ets-ogcapi-processes10/issues/92