Implementing OGC API for processes

[cehbrecht]

We want to implement the client part for the upcoming Rest API for Web Processing Services: OGC API - processes

We can use this issue to collect ideas and resources.

[huard]

Is there value in using the SDK automatically generated by https://app.swaggerhub.com/apis/geoprocessing/WPS/1.0-draft.5 ?

Export > Client SDK > Python

A lot of the boilerplate stuff seems to be handled already, but it's not clear how this meshes with the existing client implementation for features, collections, etc.

[cehbrecht]

Spec:

• https://ogcapi.ogc.org/processes/
• https://docs.opengeospatial.org/DRAFTS/18-062.html
• https://app.swaggerhub.com/apis/geoprocessing/WPS-all-in-one/1.0-draft.6-SNAPSHOT

Reference Implementations: https://github.com/opengeospatial/ogcapi-processes/blob/master/implementations.adoc

Could start with pygeoapi as a test service.

Install locally: https://docs.pygeoapi.io/en/latest/installation.html

Example Execute Request:

http://localhost:5000/processes/hello-world/jobs

Post request with payload:

{
"inputs": [
{"id": "name", "value": "hello"}
]
}

[cehbrecht]

Collecting test data for offline tests:

• 52north: #748

[cehbrecht]

s there value in using the SDK automatically generated by https://app.swaggerhub.com/apis/geoprocessing/WPS/1.0-draft.5 ?

maybe as source of inspiration? Could copy+paste code?

[fmigneault]

Do we want to add getJson methods and other 'reader' classes within the existing WebProcessingService class? I find many of the items should be handled by swapping the reader. (eg: when getcapabilities is called, instead the ogc-api reader would do GET /processes and parse JSON).

[fmigneault]

For testing responses, I can open access to this one for the duration of the code-sprint: https://ogc-ades.crim.ca/ADES The core OGC-API processes routes should be very close to the reference API with real processes.

[huard]

@fmigneault Could you create static response files similar to https://github.com/geopython/OWSLib/pull/748 ?

Even better would be a function to automatically fetch static response files from the server, so that when the server gets an update, we can refresh the static test response files in the OWSLib repo.

[cehbrecht]

Links from chat:

• python classes generated from json schema: pydantic
• a wps/processes implementation using pygeoapi: https://github.com/eurodatacube/pygeoapi-kubernetes-papermill

[fmigneault]

@huard They are here: https://ogc-ades.crim.ca/ADES/json But only a flat list (no $ref).

Generated from : https://github.com/crim-ca/weaver/blob/master/weaver/wps_restapi/swagger_definitions.py Which decorate corresponding routes in https://github.com/crim-ca/weaver/blob/master/weaver/wps_restapi

[cehbrecht]

Do we want to add getJson methods and other 'reader' classes within the existing WebProcessingService class? I find many of the items should be handled by swapping the reader. (eg: when getcapabilities is called, instead the ogc-api reader would do GET /processes and parse JSON).

There will a new implementation for the rest-apis: https://github.com/geopython/OWSLib/tree/master/owslib/ogcapi

I try to make a rough start ...

[cehbrecht]

I have opened a PR with a prototype implementation: https://github.com/geopython/OWSLib/pull/750

It is on branch in owslib: https://github.com/geopython/OWSLib/tree/feature-749-ogcapi-processes

Lots missing and not working yet 😄

@fmigneault @huard do you have commit rights on owslib? We can add you to the developers.

[fmigneault]

I don't have them.

[cehbrecht]

@fmigneault

I don't have them.

now you have :smile:

[cehbrecht]

started to look at pydantic for json schema model: #755

[cehbrecht]

updated ogcapi.processes using pydantic schema model #758.

Not too convinced yet :)

It is quite some code and currently the schema is not fixed. The schema should already be defined in the specs (using json/yaml ?). For validation one might use jsonschema?

We could instead have utility classes for inputs and outputs. One should always be able to use json for input and outputs. The optional utility classes can simplify the creation of inputs and reading the outputs with python methods. They don't need to map 1-to-1 with the schema.

@huard @fmigneault @tomkralidis thoughts?

[huard]

I like the fact that the base classes have a one-to-one correspondence with the schema. I agree that while the schema is still a moving target, it may be tricky to keep things aligned, but my feeling is that for the long term, this is a solid foundation for a robust client.

My sense is that we should not try too hard to get the client "working" until the standard is frozen. In the meantime, it may provide useful feedback to folks working on the standard.

What do you mean by utility classes ?

[cehbrecht]

What do you mean by utility classes ?

Just helper classes to build a json document for the execute ...

[huard]

Agree this is useful, but I thought we would do that in birdy, and keep OWSLib as the low-level client.

[tomkralidis]

In general, I'm not a fan of auto-generation of code. While they are very attractive on the surface, they are also very hard to sort out artifacts, differences and issues once you dive in a bit deeper. At which point you may discover that the you haven't really saved much time. As specifications evolve, what does the lifecycle look like to make updates? Do you run the autogeneration again? What about resolving the differences between new standard and the changes that were made in between the standard and monkey-patching the code to fit?

[cehbrecht]

Looking at the schema is clearer: https://github.com/opengeospatial/ogcapi-processes/tree/master/core/openapi/schemas

The examples are somehow confusing: https://github.com/opengeospatial/ogcapi-processes/tree/master/core/examples/json

[fmigneault]

I have found examples that are definitely not aligned with the schema themselves. I also find that using existing YAML file references to validate the schema is better than trying to redefine them here (they will be out of sync fast). There is no advantage to re-code the classes forming the same schema structure.

Also, as mentioned in previous comments, JSON makes it very natural to use in python. Contrary with XML where parsers were quite useful to generate the documents, simple json,load/json.dumps does perfectly the job to obtain the python types.

The execute body-builder is a good idea. If we can simplify it to something like follow, it will be convenient to use:

wps = ogcapi.Process("<url>")
data_inputs = {"input-file": "<file>", "input-value": 123}
assert process.validate_execute(id="process", data_inputs)
exec = process.execute(id="process", data_inputs) 
status = exec.monitor()

[cehbrecht]

We don't need to make a decision on the implementation yet ... can still explore the possibilities and see how they feel like 😄

[tomkralidis]

Agree/+1.