search

geopython / pywps

PyWPS is an implementation of the Web Processing Service standard from the Open Geospatial Consortium. PyWPS is written in Python.
https://pywps.org
MIT License
175 stars 117 forks source link

Feature Request: use OWS Context (an "external" WPS process description) #322

Open cehbrecht opened 6 years ago

cehbrecht commented 6 years ago

Description

The OGC OWS Context can be used to describe a WPS process. The WPS process description would then be independent of the WPS service implementation. A processing tool could be bundled with its WPS process description (OWS Context document) and picked up by a any WPS service which is able to read OWS Context documents. Processes would become plugins with a plugin description which can be used by different WPS serivces without changes.

Steps to Reproduce

Additional Information

huard commented 6 years ago

Will this facilitate the translation of service description to other languages ? Issue #64

cehbrecht commented 6 years ago

@huard To be honest I haven't looked deeply in the OWSContext. My understanding of it comes from the talks with Tom.

In the testbed they have the example of a dockerized application which is executed by a WPS service. The application uses on OWSContext document for its description (inputs/outputs). The container is executed like a command line script with defined inputs ... outputs are collected afterwards. So, the WPS service can call anything ... it is just a command-line script.

Don't know how the logging and status information (progress) is handled.

tomLandry commented 6 years ago

This will be one of the first things we'll design. As DKRZ, Ouranos and Tom Kralidis are TB-14 observers, I expect the information will make its way pretty nicely. I'll still have to double-check with OGC. David Byrns will probably lead the design, with my help. Kick off was awesome by the way!

allixender commented 6 years ago

Hi all, I have been working with OWSContext/OWC for a while now, but in with Scala/Play Framework and TypeScript/Angular (https://github.com/ZGIS/smart-owc-geojson and )

I started in a project where we also aim to evaluate SensorML2 vs WPS XML vs OWC as complex process descriptions for something like environmental models. So far we are lacking useful real-world implementation of a WPS to do that. I used a similar approach some time back with WMC and 52N WPS but that was only some hard-coded mess from my side.

I am keen to contribute, because I won't build a WPS in Scala from scratch any time soon, but the 52N Java Servlet implementation is not my favourite either. I had great experiences with Tom Kralidis and PyCSW so I might as well just stcik eith Python and give PyWPS a shot. I am at the Univisersity of Tartu, and we are not yet OGC members but I am bringing this on the way. We have seen some issues with the definitiuons of OWC Geojson Encoding but couldn't really bring them towards the OGC working group for OWC (https://allixender.blogspot.co.at/2017/07/ogc-ows-geojson.html and https://github.com/ZGIS/smart-owc-geojson/issues/4).

Happy to pick that up with you guys, if possible.

How shall we get started?

jachym commented 6 years ago

Hi, OWC seems something we need to support and seems to bring the (at least to me) important topic "containerization" of running jobs.

@allixender I think, if you are interested in this topic, try please to have a look at waht @lazaa32 is doing with docker support, that could be one of the ways.

And if I understand correctly, we need also OWC parser class (which should be implemeted in OWSLib imho).

PyWPS/pycsw/OWSLib are sibling projects, if you like the one, you should not have problem with the other.

But having real-world WPS implementations is hard. I thing @cehbrecht has some really good working. I would be happy to help with deployment of some others.

allixender commented 6 years ago

Hi, yes seems to make sense to implement OWC encodings (@tomkralidis AtomXML and? OWC GeoJSON in OWClib first). Yes, I'll look it up.

allixender commented 6 years ago

I created a ticket in OWSlib to implement the OWS Context types, which then could be used here in PyWPS: Implement OGC OWS Context format https://github.com/geopython/OWSLib/issues/441 ... and implemented most fields https://github.com/geopython/OWSLib/pull/483

tomLandry commented 6 years ago

Link to public code repository from EOC hackathon: https://github.com/opengeospatial/EOEPHackathon2018

christophenoel commented 6 years ago

Hello,

Could you please provide you view about why the WPS process description would be independent of the WPS service implementation ? I don't understand this point.

The WPS already defines a nice standard for defining the Process interface which is that Process Description. Why don't you put the OWS Context inside the existing Process Description ?

Also, OWS Context is not (especially) written to describe inputs/outputs, so did you define a standard for defining such interface ? What is the advantage of creating an additional specification to the existing Process Description ?

Thanks a lot in advance for the clarification(s). Christophe Noel.

tomLandry commented 6 years ago

Hello @spacebel. Are your question formulated to the implementer @allixender? As for myself, I only pointed out this timely work to the OGC testbed, and vice-versa. I'd like to try out the implementation and I welcome clarifications and contextualisation!

christophenoel commented 6 years ago

Hello Tom, all answers are welcome :) I try to understand the approach. Thanks !

tomLandry commented 6 years ago

A bit more context for my side then. Very early in the testbed, I was expecting OWS Context to be quickly selected and used by TB-14 participants - an educated guess. As you know, this was not exactly where discussions have been focussed on in the last few months. Aside from providing very high level coordination for this particular work, CRIM was not involved in the implementation in any form.

As you may know, as Testbed-13 participants, CRIM only very slightly touched OWS Context. I refer to your experience and lessons learned at Spacebel on this. Depending on the state of discussions, CRIM could test or adapt implementations for the benefit of the testbed - if possible. Note that we are PyWPS users, so we welcome these advances and would like to contribute.

allixender commented 6 years ago

Hi @spacebel , so my background thinking is coming from the way to bundle resources with OWS Context. I totally agree that WPS Process description XML is there and it is specific, appropriate and works. However, over many years various approaches have been tried to semantically enrich WPS process descriptions in particular for input data requirements. And secondly chaining/cascading WPS calls in terms of workflow also has various approaches, but mostly you need an external workflow orchestrator. So my thinking is looking into using OWS Context actually on top of WPS, not necessarily completely discarding WPS XML, but using OWS Context as one supported input format (like GML etc) and have a WPS process with logic of pulling the referenced datasets in the OWS Context and use with specified WPS process. And thus, could also call a related subordinate WPS process and use the results in the current process (as referenced through the OWS Context). For this type of resource-referencing interoperable and exchangeable descriptive format I'd like to test OWS Context (WPS XML always only really stays on the WPS). Another variant to supersede or extend current WPS XML situation is to test SensorML2.0 for process desiption. Also a mich richer feature set to describe processes with input and outputs. And SensorML is also used in other services, e.g. OGC SOS and beyond to describe things and interactions between sensors/processes and dataflow. Let me know what you think 😄 Cheers

christophenoel commented 6 years ago

Hello Allixender, thanks a lot for your answer ! I see you have a deep experience on the subject. I also worked a lot with WPS, WPS-T (mostly with workflow engine backends (BPEL, BPMN, CWL) and cloud/grid backends (Globus, Docker, Kubernetes). I have proposed some implementations based on 52North WPS (yes Java :p)

My opinion on the main topics are:

Embedding the metadata in the Process Description has some advantages:

My two cents ;). Christophe Noel.

jachym commented 6 years ago

Just noting, that this is slightly out of my scope, so if you feel I should contribute somewhere or do something, which would help with PyWPS development, please write me directly at jachym.cepicky at gmail

tomLandry commented 6 years ago

Hello everyone. I've been authorized to share updated reference OWS Context files provided by CubeWerx in OGC Testbed-14. You can find them here: http://www.pvretano.com/Projects/tb14/ApplicationPackage/owc/

At this point, I'd like to point out that CRIM will - in short term - follow its initial plan of describing applications using JSON. This is not a technical or architectural decision, more a capacity and timing decision. There is a suggestion in the OGC testbed to use both OWS Context and JSON descriptors for applications packages (bundles). Thus, CRIM will therefore complete JSON descriptors in an end-to-end worlflow before trying OWS Context implementation presented here.

In the meantime, kudos to Spacebel and CubeWerx for valuable inputs on OWS Context, and obviously, kudos to @allixender for provinding this timely initial implementation in OWSLib!