search

geopython / pygeoapi

pygeoapi is a Python server implementation of the OGC API suite of standards. The project emerged as part of the next generation OGC API efforts in 2018 and provides the capability for organizations to deploy a RESTful OGC API endpoint using OpenAPI, GeoJSON, and HTML. pygeoapi is open source and released under an MIT license.
https://pygeoapi.io
MIT License
459 stars 250 forks source link

Adding support for processing microservices #1671

Open herzogrh opened 1 month ago

herzogrh commented 1 month ago

Is your feature request related to a problem? Please describe. Right now, processes for the OGC API Processes can only be implemented in Python.

Describe the solution you'd like It would be great to connect the pygeoapi with processing microservices via eg. Websockets connections. In that way, it is possible to execute processes in other environments than in Python, but still make them accessible through pygeoapi's OGC API processes and job management. As an example, one could connect a node.js processing microservice with pygeoapi and run Insight Maker simulations. Another benefit is that one can dynamically add and remove processing microservices without having to recompile pygeoapi.

Once a microservice establishes a connection to the pygeoapi endpoint, it registers a process that is then added to the list of processes in pygeoapi. Whenever an execution request is made by a client, the proposed feature will forward the inputs through the eg. websocket connection and the microservice will start the calculation. Once the results are calculated, the microservice transmits the results back to be stored in the job management.

Benefits are, as pointed out by @geotom in #1484:

Describe alternatives you've considered Implementing a job management and OGC API Processing complient endpoints in every single programming language, which sounds like an awful lot of work to me.

Additional context

In the urban digital twin project Connected Urban Twins we identified the use case of multi-dimensional simulations running on various different modelling paradigms and software stacks. Therefore it is necessary to easily spin up and deploy various processing microservices via standardised APIs such as the OGC API Processes. As pygeoapi is a great implementation of OGC API processes and provides both an interface and an integrated job management, the feature to connect Websocket processing microservices would be greatly appreciated. We implemented a rough version ourselves which can be found here as a fork of the pygeoapi: Urban Model Server

geotom commented 1 month ago

Great initiative @herzogrh

I have a lot of ideas and past implementation experience for WPS. Making the microservice job management agnostic is definitely something I would prefer. A plugin could ship a default protocol/API leaving it open to users implement their own transports. I have already had a look to https://github.com/airtai/faststream.

Other brokers/queues I would like to see or might be useful are

Additionally I would love to see

tomkralidis commented 1 month ago

Keeping mind the pygeoapi focus on OGC API standards, note that Pub/Sub support is coming via the forthcoming OGC API - Pub/Sub efforts. This will essentially allow pygeoapi to publish/subscribe to a broker, which can then be tied loosely enough from/to/with other services.

herzogrh commented 3 weeks ago

Thanks for the info @tomkralidis! If I understand correctly, OGC API Pub/Sub would be the preferred way of subscribing to a processing microservice. I suppose that still, there would have to be a link between Pub/Sub and API Processes. So that if subscribed to a broker, pygeoapi would publish the process registered by the broker via its API processes endpoint.

tomkralidis commented 3 weeks ago

Sure. I don't have an overview/picture of the workflow, but in OGC API - Pub/Sub, pygeoapi would basically publish messages to a Pub/Sub broker (MQTT, AMQP, WebSockets, Kafka, etc.) that would trigger additional workflow.