search

opengeospatial / ets-wmts10

WMTS Test suite
Other
1 stars 3 forks source link

Improve performance of the test #2

Closed rjmartell closed 9 years ago

rjmartell commented 10 years ago

What kind of speedup should we see (on the "reworked" branch)? If I choose to run all tests for the "GET" profile only, it still takes approx. 24 min to complete a test run that includes 72 tests of which almost half are skipped.

wmts-get-results

While this is definitely quicker than before, is it fast enough? Are there some very slow tests that account for most of the time?

bermud commented 10 years ago

Hi Sebastian, can you please confirm the details of how you run the test and what passed?

bierett commented 10 years ago

I have execute the WMTS TestSuite against our CITE test environment (with deegree webservices 3.4). I have seen that the most test were skipped, because of failed prerequisites and incorrect response.

For example: Valid KVP GET GetCapabilities request with the optional AcceptVersions parameter. It is expected that the service will respond with HTTP status code 400. But our service respond with HTTP status code 200.

Message of TeamEngine Test skipped: WMTS server version 1.0.0 did not return a correct ows:ExceptionReport and HTTP status in response to request with an acceptversions parameter value of 1.1.0,1.2.0,so Accept Versions considered to not be implemented.

bierett commented 10 years ago

I have create a list (see GIST) of all tests which failed or skipped against deegree webservices 3.4.

I see only three tests which are failed / skipped and caused by our fix:

I will have a further look at this. For all other issues I will create tickets in the deegree github issue tracker.

bierett commented 9 years ago

I had a further look on the ETS with deegree webservices 3.4-pre15.

For profile GET Binding

I have check all tests which are failed. Most of them are failed because of bugs in deegree. Some tests are failed because of not-existing encoding of string, which are used in the URI. This bug is fixed with https://github.com/lat-lon/ets-wmts10/commit/59520ad1cbb03bf02bd298b4290c5e76d2af472c.

I have create the following issues. Only for test wmts:Server.KVP.GET.HTTP.Mandatory I have to check if the assertion is correct. For the moment I have not found any hint in the WMTS spec.

bierett commented 9 years ago

I have create a Pull Request https://github.com/opengeospatial/ets-wmts10/pull/4 with all changes.

rjmartell commented 9 years ago

It took approx. 20 min to run only the GET profile with these results:

wmts-results

I noticed that some tests are running very slowly. For example, according to the test log for Server.KVP.GET.InvalidQueryString the response was received in 0.3 s but schema validation (exception.validation.elapsed) took 17.6 s (!).

Is it worth trying to speed up the very slow tests? Of the 4 tests in the CommonOperations.Mandatory.KVP.GET group where schema validation was performed, every validation episode took about 17 s. This is terribly slow.

bermud commented 9 years ago

Hi Richard, Do we know why the schema validation is taking soo long? Is it trying to get external hosted schemas, or ?

rjmartell commented 9 years ago

That seems to be the case for exception reports. For example, this message is logged:

Validating ows:ExceptionReport against its XML Schema http://schemas.opengis.net/ows/1.1.0/owsExceptionReport.xsd

Note that only the WMTS schemas are provided as local resources. I opened #6.

rjmartell commented 9 years ago

Merged branch 'reworked-ets' to master (see commit f48b4dfd79ea12bb8bc9d33ce065a7069d3c5529). The speedup is significant (S >= 2).