search

Open-EO / openeo-test-suite

Test suite for validation of openEO back-ends against the openEO API specification
Apache License 2.0
0 stars 2 forks source link

Feature: WP4 General API compliance validation #45

Closed GeraldIr closed 9 months ago

GeraldIr commented 9 months ago

NOTE: because I kept working on my own fork of the openeo-test-suite repo, this also includes the original changes from WP3, these can be ignored for the purpose of review (and should probably be merged seperately and not rely on this PR alone).

Add: general API compliance tests to the openeo-test-suite as requested in https://github.com/Open-EO/openeo-test-suite/issues/20

The tests cover a range of different scenarios and endpoints and the openapi-core library is used to check the validity of responses from the API against the openapi specification. (The one included is the newest at version 1.2.0).

The files included are basic payloads, api spec (openapi.json), a utility python file and the test file containing the full range of tests.

Also note that actual requests are sent to the API, I have put skip marks on any tests that need a process graph to run through so that runtimes are not too long. Any created batch jobs, UDPs or other objects are cleaned up after every test.

Usage:

pytest src/openeo_test_suite/tests/general --html=reports/general-compliance.html --openeo-backend-url=openeo-dev.eodc.eu

GeraldIr commented 9 months ago

A personal note for reviewers considerations, this code is not cleaned up. It does work and I did my best to integrate it with the existing infrastructure, but keep in mind that big refactors or cleanups might take some additional time. For now I suggest focusing on the functional aspects and any missing endpoints/scenarios and if time allows I will look into cleaning up in the future.

m-mohr commented 9 months ago

Conducted a very high-level review, didn't look into details yet. But it should already help to avoid some potential issues. For a more detailed review I'm waiting until the changes from #44 are not part of this PR changelist any longer.

soxofaan commented 9 months ago

quick note: can you move the submodule to https://github.com/Open-EO/openeo-test-suite/tree/main/assets (which already has the openeo-processes submodule)?

GeraldIr commented 9 months ago

quick note: can you move the submodule to https://github.com/Open-EO/openeo-test-suite/tree/main/assets (which already has the openeo-processes submodule)?

Yes of course!

GeraldIr commented 9 months ago

quick note: can you move the submodule to https://github.com/Open-EO/openeo-test-suite/tree/main/assets (which already has the openeo-processes submodule)?

@soxofaan This is done. I've also updated the readme.