search

opengeospatial / teamengine

TEAM Engine (Test, Evaluation, And Measurement Engine) is an engine for testing web services and other resources written in JAVA. It executes test scripts written in Compliance Test Language (CTL), TestNG and other languages. It is lightweight and easy to run as a command line or to setup as a service. It can be used to test any type of service or encoding. It is also the official tool used by the Open Geospatial Consortium (OGC) for compliance testing.
Apache License 2.0
45 stars 41 forks source link

REST API: Test run against CTL based test suite does not return test result #548

Closed dstenger closed 2 years ago

dstenger commented 2 years ago

Describe the bug When running a CTL based test suite via REST interface and setting HTTP header Accept: application/rdf+xml, the returned response contains elements testsSkipped, testsPassed and testsWarning filled with 0. Thus, the report does not provide the test result.

However, when monitoring the log file, it seems that the tests were executed.

At the bottom, I added an example response of WMS 1.3 test suite executed via REST interface.

This bug occurs with TEAM Engine v5.5.

To Reproduce Steps to reproduce the behavior:

  1. Start WMS 1.3 test suite with TEAM Engine v5.5.
  2. Execute: GET http://localhost:8081/teamengine/rest/suites/wms13/run?capabilities-url=https%3A%2F%2Fcite.deegree.org%2Fdeegree-webservices-3.4.16%2Fservices%2Fwms130%3Fservice%3DWMS%26request%3DGetCapabilities with Accept: application/rdf+xml.
  3. View the response.

Expected behavior The report shall contain the correct test result.

Additional context Example response when executing WMS 1.3 test suite via REST interface:

<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:dct="http://purl.org/dc/terms/" xmlns:cnt="http://www.w3.org/2011/content#" xmlns:cite="http://cite.opengeospatial.org/" xmlns:earl="http://www.w3.org/ns/earl#" xmlns:http="http://www.w3.org/2011/http#">
   <earl:TestSubject rdf:about=""/>
   <earl:Assertor rdf:about="https://github.com/opengeospatial/teamengine">
      <dct:title xml:lang="en">OGC TEAM Engine</dct:title>
      <dct:description xml:lang="en">Official test harness of the OGC conformance testing program (CITE).</dct:description>
   </earl:Assertor>
   <cite:TestRun>
      <cite:areCoreConformanceClassesPassed rdf:datatype="http://www.w3.org/2001/XMLSchema#boolean">true</cite:areCoreConformanceClassesPassed>
      <cite:requirements>
         <rdf:Seq/>
      </cite:requirements>
      <cite:testsContinue rdf:datatype="http://www.w3.org/2001/XMLSchema#int">0</cite:testsContinue>
      <cite:testsNotTested rdf:datatype="http://www.w3.org/2001/XMLSchema#int">0</cite:testsNotTested>
      <dct:title>Web Map Service (WMS) 1.3.0</dct:title>
      <cite:testSuiteType>ctl</cite:testSuiteType>
      <cite:testsSkipped rdf:datatype="http://www.w3.org/2001/XMLSchema#int">0</cite:testsSkipped>
      <cite:testsPassed rdf:datatype="http://www.w3.org/2001/XMLSchema#int">0</cite:testsPassed>
      <cite:testsWarning rdf:datatype="http://www.w3.org/2001/XMLSchema#int">0</cite:testsWarning>
      <cite:inputs>
         <rdf:Bag>
            <rdf:li rdf:parseType="Resource">
               <dct:title>acceptMediaType</dct:title>
               <dct:description>application/rdf+xml</dct:description>
            </rdf:li>
            <rdf:li rdf:parseType="Resource">
               <dct:title>sourcesId</dct:title>
               <dct:description>OGC_Conformance Test Suite - OGC Web Map Service 1.3.0_1.29-SNAPSHOT</dct:description>
            </rdf:li>
            <rdf:li rdf:parseType="Resource">
               <dct:title>capabilities-url</dct:title>
               <dct:description>https://cite.deegree.org/deegree-webservices-3.4.16/services/wms130?service=WMS&amp;request=GetCapabilities</dct:description>
            </rdf:li>
            <rdf:li rdf:parseType="Resource">
               <dct:title>updatesequence</dct:title>
               <dct:description>auto</dct:description>
            </rdf:li>
            <rdf:li rdf:parseType="Resource">
               <dct:title>sessionId</dct:title>
               <dct:description>s0004</dct:description>
            </rdf:li>
            <rdf:li rdf:parseType="Resource">
               <dct:title>basic</dct:title>
               <dct:description>basic</dct:description>
            </rdf:li>
            <rdf:li rdf:parseType="Resource">
               <dct:title>logDir</dct:title>
               <dct:description>/root/te_base/users/ogctest/rest</dct:description>
            </rdf:li>
         </rdf:Bag>
      </cite:inputs>
      <dct:created>2022-07-20T10:30:55.110Z</dct:created>
      <cite:testsInheritedFailure rdf:datatype="http://www.w3.org/2001/XMLSchema#int">0</cite:testsInheritedFailure>
      <cite:testsFailed rdf:datatype="http://www.w3.org/2001/XMLSchema#int">0</cite:testsFailed>
      <cite:testsBestPractice rdf:datatype="http://www.w3.org/2001/XMLSchema#int">0</cite:testsBestPractice>
   </cite:TestRun>
</rdf:RDF>