search

opengeospatial / ets-ogcapi-features10

Public Repository for the OGC API - Features Compliance Test Suite
Other
16 stars 6 forks source link

Simplify detecting test failures #242

Open captaincoordinates opened 2 weeks ago

captaincoordinates commented 2 weeks ago

Is your feature request related to a problem? Please describe. It is not clear to me how to properly identify CITE test failures in an automated environment. I believe it's currently necessary to parse the XML results file to identify failures. This GitHub Actions workflow file appears to show how tests can be run as part of a CI pipeline, but does not appear to fail following a negative test result. It also appears that this action has never been run, and I cannot find a file called parsetestresults.py anywhere (even though I'm not sure it would be useful).

Ultimately I would like to run CITE compliance tests as part of a CI test suite so that compliance failures can block an automated deployment.

Describe the solution you'd like java -jar ets-ogcapi-features10-1.8-SNAPSHOT-aio.jar... should vary its exit code based on test results. If any tests fail it should exit non-zero.

Describe alternatives you've considered Better documentation of how to identify test failures in an automated environment.

dstenger commented 2 weeks ago

Thank you for the feedback.

Please see chapter "Output formats" of https://opengeospatial.github.io/teamengine/users.html for a description of the different output formats.

In general, there are different ways how to execute the test suite (e.g. REST interface or command line). Using the *aio.jar is one of those options.

My proposal is to parse the EARL output to retrieve the passed and failed tests. As far as I know, there is unfortunately no script provided by the CITE team which is executing the parsing. One reason is that this can be done with any programming language. If you are writing such a script, we would really appreciate it if you can share your work.

captaincoordinates commented 2 weeks ago

Thanks, I'll post back here if I put a script together for this