Improve organisation of tests

opengeospatial / ets-wms11

Executable Test Suite for WMS 1.1.X

0 stars 5 forks source link

Improve organisation of tests #58

Closed lgoltz closed 3 years ago

lgoltz commented 6 years ago

As described in https://github.com/opengeospatial/ets-wms11/issues/53#issuecomment-378842640 the organisation of the tests is confusing. On the main page of the WebGUI one of the three profiles can be select:

NONE - Do not certify that the server meets requirements for any test profile
BASIC - Certify that the server meets all requirements for the BASIC profile
QUERYABLE - Certify that the server meets all requirements for the QUERYABLE profile

The executed tests differs depending on the selected profile:

NONE: only tests assigned to NONE are executed
BASIC: tests assigned to NONE and BASIC are executed
QUERYABLE: tests assigned to NONE and QUERYABLE are executed

If QUERYABLE is selected the BASIC tests are not executed (and in the new HTML report only NONE is marked as basic conformance class).

This must be discussedin the context of the new HTML report.

ghobona commented 4 years ago

The standard says:

2.1 Conformance classes and requirements

This International Standard defines two conformance classes, one for a basic WMS, and the other for a queryable WMS. Each has two subclasses, one for clients and the other for servers.

2.2 Basic WMS

A basic WMS shall support the basic service elements (see Clause 6), the GetCapabilities operation (see 7.2), and the GetMap operation (see 7.3). To conform to this International Standard, a basic WMS shall satisfy the requirements of A.1 of the Abstract Test Suite in Annex A.

2.3 Queryable WMS

A queryable WMS shall satisfy all the requirements for a basic WMS, and shall also support the GetFeatureInfo operation (see 7.4). To conform to this International Standard, a queryable WMS shall satisfy all requirements of the Abstract Test Suite in Annex A.

keshavnangare commented 4 years ago

As per discussion, I have updated the test behaviour as per user inputs and removed the NONE profile from UI. The following behaviour is updated:

BASIC: Only Basic profiles test will be executed.
QUERYABLE: Basic + Queryable profiles will be executed.

Please find the changes in the #77 PR.

dstenger commented 3 years ago

@keshav-nangare Currently, the integration-tests are failing when running the build with mvn clean install site docker:run -Pintegration-tests,docker. It looks like that the start of wms:wms-no-profile was removed in the pull request. However, the tests shall still be executed when selecting BASIC or QUERYABLE. Just the option to execute NONE only shall be removed.

keshavnangare commented 3 years ago

@dstenger

I have added start to keep the NONE profile tests execution. Also, I updated integration tests pass count and iut url. Please find the changes in the same PR.