search

opengeospatial / ets-wms13

Repository for the Exectutable Test Suite for WMS 1.3.0
Apache License 2.0
2 stars 4 forks source link

Tests are always run exhaustively #91

Open jyrkka opened 2 years ago

jyrkka commented 2 years ago

If I have understood correctly, looking at the CTL XML-file I see there should be possibility to run tests non-exhaustively. When running test via web application should there be a some form which would ask me if I want to test each Layer/CRS combination?

https://github.com/opengeospatial/ets-wms13/blob/dc7025c463f439094dbeddafec4e018c6bf649f1/src/main/scripts/ctl/main.xml#L271

And when running test via rest interface, 'exhaustive' variable value is set to 'no', but despite that the test are still run exhaustively.

https://github.com/opengeospatial/ets-wms13/blob/dc7025c463f439094dbeddafec4e018c6bf649f1/src/main/scripts/ctl/main-auto.xml#L155

Have I misunderstood something or is this expected behviour?

dstenger commented 2 years ago

Thank you for reporting.

Following part of the code is skipped

https://github.com/opengeospatial/ets-wms13/blob/dc7025c463f439094dbeddafec4e018c6bf649f1/src/main/scripts/ctl/main.xml#L271

when running it as a servlet. This is evaluated by the method te:isWeb().

Thus, this variable has no function when using it with the Web Browser Interface or the REST Interface. Using the CLI will probably consider this variable.

Is there a specific reason you need this functionality?

jyrkka commented 2 years ago

The reason why I would need that is because our system contains thousands of different supported CRS and the time to run the tests is over half an hour. And also at the end of the test suite at the moment i get OutOfMemory error when it tries to create the final report. The OOM can be surely fixed somehow but, when running these test as a part of a CI-pipe, i would like to run these test non-exhaustively also.

I also noticed this: https://github.com/opengeospatial/ets-wms13/blob/dc7025c463f439094dbeddafec4e018c6bf649f1/src/main/scripts/ctl/getmap.xml#L75

Should that condition be other way around? That is the main reason why test are taking so long when the tests are run via rest. As I said in the previous message, the 'exhaustive' variable value is set to 'no' in the main-auto.xml. And with that if condition, suite will call 'each-crs' test`

dstenger commented 2 years ago

I agree that this line might introduce a bug. However, in the recent years we did not actively use or modify this parameter/ functionality. So, it seems there should be a complete rework regarding the exhaustive parameter.

Did you set up TEAM Engine locally? If that is the case, can you please try to modify this line and check if it solves your problem?

jyrkka commented 2 years ago

Yes, I tried it with local TEAM Engine installation and changing that line solved the problem.