InheritedFailure and error in prerequisite checks for several optional features

[matsoo]

When running the WMTS-tests on the beta site, several of the test groups generate a inherited failure even if all the subtests pass or are skipped. The tests groups have in common that they all test optional functionality that is not implemented on the server and is skipped. I have only tested GET binding and REST binding as our server does not implement the other bindings, so they may also have the same problem.

You can see the behaviour when using the following service: http://192.71.92.21/MapService/wmts.axd/Vmap0WMTS?Service=WMTS&request=GetCapabilities

When looking at the log for the test groups I can see the following errors:

wmts:GetCapabilities.Implemented.KVP.GET: ERROR: prereq name: Server.KVP.GET.GetCapabilities.Response.No.Layers.OtherSource.Prerequisites met prereqs false does not match selected name Server.KVP.GET.GetCapabilities.Response.No.Layers.OtherSource.Selected selected no value: ERROR: prereq name: Server.KVP.GET.GetCapabilities.Response.Dimension.Identifiers.Unique.Prerequisites met prereqs true does not match selected name Server.KVP.GET.GetCapabilities.Response.Dimension.Identifiers.Unique.Selected selected no value:

wmts:GetTile.Implemented.KVP.GET: ERROR: prereq name: Server.KVP.GET.GetTile.Missing.Optional.Prerequisites met prereqs false does not match selected name Server.KVP.GET.GetTile.Missing.Optional.Selected selected no value: ERROR: prereq name: Server.KVP.GET.GetTile.Invalid.SampleDimension.Name.Prerequisites met prereqs false does not match selected name Server.KVP.GET.GetTile.Invalid.SampleDimension.Name.Selected selected no value:

wmts:GetFeatureInfo.Implemented.KVP.GET: ERROR: prereq name: Server.KVP.GET.GetFeatureInfo.Invalid.Tile.Row.No.TileMatrixSetLimits.Prerequisites met prereqs true does not match selected name Server.KVP.GET.GetFeatureInfo.Invalid.Tile.Row.No.TileMatrixSetLimits.Selected selected no value: ERROR: prereq name: Server.KVP.GET.GetFeatureInfo.Invalid.Tile.Col.No.TileMatrixSetLimits.Prerequisites met prereqs true does not match selected name Server.KVP.GET.GetFeatureInfo.Invalid.Tile.Col.No.TileMatrixSetLimits.Selected selected no value: ERROR: prereq name: Server.KVP.GET.GetFeatureInfo.Missing.Optional.Prerequisites met prereqs false does not match selected name Server.KVP.GET.GetFeatureInfo.Missing.Optional.Selected selected no value: ERROR: prereq name: Server.KVP.GET.GetFeatureInfo.Invalid.SampleDimension.Name.Prerequisites met prereqs false does not match selected name Server.KVP.GET.GetFeatureInfo.Invalid.SampleDimension.Name.Selected selected no value:

wmts:GetTile.Implemented.RESTful: ERROR: prereq name: Server.RESTful.GetTile.Missing.Optional.Prerequisites met prereqs false does not match selected name Server.RESTful.GetTile.Missing.Optional.Selected selected no value:

wmts:GetFeatureInfo.Implemented.RESTful (View Details): ERROR: prereq name: Server.RESTful.GetFeatureInfo.Invalid.Tile.Row.No.TileMatrixSetLimits.Prerequisites met prereqs true does not match selected name Server.RESTful.GetFeatureInfo.Invalid.Tile.Row.No.TileMatrixSetLimits.Selected selected no value: ERROR: prereq name: Server.RESTful.GetFeatureInfo.Invalid.Tile.Col.No.TileMatrixSetLimits.Prerequisites met prereqs true does not match selected name Server.RESTful.GetFeatureInfo.Invalid.Tile.Col.No.TileMatrixSetLimits.Selected selected no value:

[bermud]

I also added the issue in TE, with screenshots and logs https://github.com/opengeospatial/teamengine/issues/141.

@dstenger, please let us know if this is an issue of the WMTS test or TE.

[bermud]

@dstenger, have you found the issue?

We have found the issue might be in the test. TE is behaving correctly.

The problem is with mismatches on the functions parameters.

For example: In the wmts test suite one test, in the wmtsFunction.xml file, wmts:CommonOperations.Mandatory.KVP.GET has a function wmtsFunctions:invokeSelectedTests.

This function has the variables kvpGetCapabilitiesURL and postGetCapabilitiesURL. The variable postGetCapabilitiesURL tries to get the ows:Post/@xlink:href link from the capability URL, but the ows:post element is not present in the capability url element.

It seems that the ows:POST element is not available to the test.

[dstenger]

But why should "Test wmts:GetCapabilities.Implemented.KVP.GET" fail then and " Test wmts:GetCapabilities.Mandatory.KVP.GET" is passing?

[rjmartell]

The WMTS suite uses both the ctl:notTested (2) and ctl:skipped (3) results. The former is supposed to indicate that "a test was not performed because it was not selected for execution", whereas the latter indicates that "a test was skipped because the capabilities being tested were not implemented or some test prerequisites were not met". How do these affect the overall result assigned to a parent test (group)?

I would think they have no effect unless all other subtests were also skipped or not tested; in this case the parent should also be marked as skipped. But this is not what seems to happen. For example, choose to test the GET binding and then skip all GetFeatureInfo tests (select "None"). For the wmts:GetFeatureInfo.Implemented.KVP.GET test group the results of the constituent tests are a strange mixture of "Skipped" and "Not Tested" verdicts. However the overall verdict is shown as Failed (InheritedFailure) (5) even though no child tests were run. it should be marked as skipped (or not run, but the skip verdicts look a bit odd here since no tests were selected).

This is likely a TEAM-engine issue.

[rjmartell]

See https://github.com/opengeospatial/teamengine/issues/141