Python and/or Go?

[m-mohr]

I'm a bit confused that there are now two efforts for validation. One just got reactived (Python) and one is developed by Bernhard since some time (Go). Do we develop both or do we focus on a single one? I feel like having a single one is better and would be better with regards to saving resources? @jdries @bgoesswe

[jdries]

We just used the Python approach because our validation efforts with the Go implementation got stuck on a (probably temporary) issue, see #11. That allowed us to start fixing issues in our backend. Problem is we didn't have the go experience to do a fix in there, but wouldn't mind using it.

[bgoesswe]

Ok I will look into this issue, or why I don't get this error. Maybe it helps that we will provide a web service of the Go tool on Friday. There you can use the Go tool, without having to set it up by yourself.

[soxofaan]

Sorry, I intended to reply earlier but got sidetracked. I finally found some time to write some thoughts down.

@jdries asked me to run the validation tool against the VITO API and fix the issues that would pop up. I indeed tried the Go based tool but that didn't work out (see #11). To keep things moving I revived the Python/pytest based test suite.

After finding some initial compliance issues with simple GET endpoints, I experimented bit more with the pytest suite for more coverage and more advanced features (things that are not available in the Go based tool). For example (also see #18):

• iterate over all collections to check result of /collections/{collection_id}
• test things differently depending on the API version. For example: renamed parameter/fields, or skipping endpoints that re not available in older API version
• test /.well-known/openeo (which is special case because it should be available under domain without prefixes)
• check whether endpoints that require authentication indeed return with 401 unauthorized when no auth is provided
• check other error handling like "path not found" or "collection not found"

Based on these experiments I'd like to advocate in favour of the Python/pytest approach. (To be taken with a grain of salt as I only recently joined VITO and started working on these OpenEO projects. Also: my Go knowledge is very limited and I don't know what the Go-side of the argument has up their sleeve.)

• Config based Go tool vs pytest based test suite. A config is quite rigid compared to general programming when it comes to defining test scenarios. It is hard to define a config format that covers more advanced scenarios (e.g. sequence of GET/POST requests). And you risk a lot of fragile duplication because you have to implement things both the config file format/parsing and in the validation functions. With pytest you can just define scenario of arbitrary complexity in a single place in standard Python and use simple asserts as validation milestones
• (Long term) maintainability: within OpenEO, Python is used in quite some other projects, while Go is only used in the validator as far as I know. Python also has lower barrier to entry than Go, I think (but my opinion might be biased 😀 )
• Python is more concise to do things dynamically or automatically. For example: traverse the whole spec and run tests against (subset of) found endpoints. e.g.: check all GET endpoints that require authentication
• With pytest: easy to flag violations or problems: just assert something or raise an exception. Pytest will catch everything and report. No need for cumbersome error message building and careful passing through, like it is now done in the Go tool.

One last point in favour of Go to make things more balanced 😀. As far as I know there is no OpenApi3 request validation library for Python yet (the one for Go looks quite extensive). However, we currently we can workaround that with a combination of a JSON Schema validator and OpenApi3-to-JSON-Schema converter (like I'm also using in https://github.com/Open-EO/openeo-api/pull/194).

[bgoesswe]

So just my thoughts to the comment of @soxofaan:

I was also more into the pytest solution in the beginning, but I think we decided to look into existing validation tools to not re-invent the wheel. In general I never thought of it as a "A vs B" thing, but maybe I have a different idea of the workflow:

So I thought we will just use the Go implementation for the validation itself, but not the testing framework and more complex test cases. That's why I made a standalone tool out of it, so that everyone can use it with their own testing framework (e.g. create a pytest that iterates over all collection endpoints, but by calling the Go validation tool with the endpoints). Maybe the Go tool should get more in the direction of command line arguments and away from config files, so that there are not so many read / write operations. So having the advantage of the flexible python framework and the automatic validation of Go. But I am of course open for other ways to do it like only Python.

[m-mohr]

I myself don't have a strong opinion on a specific implementation, but just want to make sure we have a single validator we can point back-end implementers to. Also, we shouldn't implement things twice in multiple languages.