How to test view service which doesn't contain "Blue Lake" test data set?

opengeospatial / ets-wms13

Repository for the Exectutable Test Suite for WMS 1.3.0

Apache License 2.0

2 stars 4 forks source link

How to test view service which doesn't contain "Blue Lake" test data set? #90

Closed ghost closed 2 years ago

ghost commented 3 years ago

Background

In INSPIRE Community almost all service are implemented based on WMS 1.1.1 or WMS 1.3.0. Also this standard is widely used for providing services in national SDIs.

But, in order to test web services against this OGC specification using TEAM Engine, "Blue Lake" dataset has to be provided by the service. That means that Conformance Test Suite is data-dependent. Also user interaction is necessary in order to perform all test cases/test steps.

Those facts don't allow following use case:

testing productive INSPIRE WMS services manually (Team Engine UI)
perform mass validation for all productive WMS services in one metadata catalogue automatically (Team Engine API)

Questions

Is there any solution for those two use cases? Are there some other tools to support mentioned use cases?
If not, is there a plan to switch WMS Conformance Test Suite to new technology (TestNG) and new approach ("develop data-independent")?

Thanks in advance!

ghobona commented 3 years ago

Q: Is there any solution for those two use cases? Are there some other tools to support mentioned use cases?

A: If you were to rebuild the Executable Test Suite such that it did not require user interaction, you would need a Computer Vision library to evaluate the images that are returned by the server and to recognise the objects on those images. Some examples of libraries with Computer Vision capabilities include OpenCV, TensorFlow and DL4J.

Q: If not, is there a plan to switch WMS Conformance Test Suite to new technology (TestNG) and new approach ("develop data-independent")?

A: There are no plans within OGC to switch the WMS executable test suite to new technology, nor to reimplement the executable test suite using a different approach. For future reference, the OGC's Compliance Roadmap is at https://www.ogc.org/compliance/roadmap

ghost commented 3 years ago

@ghobona Thanks a lot for quick response!

One small question: Data are also some data-independent Test Cases for OGC WMS 1.3.0 Conformance Test Suite. Is it possible to address only those test cases via API?

dstenger commented 2 years ago

Unfortunately, as this is an old test suite developed in CTL, it is not possible to tackle single conformance classes via the REST API (in TestNG tests this is possible).

All options of the API are described here: https://cite.opengeospatial.org/teamengine/rest/suites/wms13/

So, basic can be enabled and disabled but this includes following tests: data-independent data-preconditions basic