JanssenBrm
commented
9 months ago Suggestion for the content:
openEO: A Guide to HTTP Requests
The following page provides instructions on utilizing the openEO API without relying on one of the provided openEO client libraries (https://openeo.org/software.html#clients). Although these libraries streamline openEO workflow development, the flexibility of openEO extends beyond the programming languages supported by these libraries. OpenEO offers a universal API that seamlessly integrates with any programming language supporting HTTP requests.
This guide walks you through the process of using the openEO API to execute batch jobs exclusively through HTTP requests. As an illustration, we'll generate variability maps for a designated area of interest using an existing openEO user-defined process available in [EOplaza](https://portal.terrascope.be/catalogue/app-details/62). Variability maps show spatial variations in crop performance within a field on a specific date, influenced by factors like soil type, hydrology, pests, diseases, or extreme weather events.
While this guide focuses on generating variability maps, the principles can be applied broadly to construct and execute any openEO workflow as a batch job.
Prerequisites
To begin with openEO, you need a valid account. The openEO Platform provides users with 1000 free credits monthly. After registering on https://openeo.cloud/, you can validate your credits on the EOplaza portal:
- Navigate to https://portal.terrascope.be/.
- Sign in and select "Edugain and social logins."
- After logging in, click the Billing tab to view your current credit score.
Now, with everything set up, we can integrate the openEO API.
Authentication
Firstly, it is important to ensure authentication and obtain a valid access token for subsequent requests. While openEO supports various authentication methods, we recommend using an OpenID Connect (OIDC) provider. In this example, we recommend using the EGI provider, which aligns with the registration process. Find a list of all supported OIDC providers and their configurations at https://openeo.vito.be/openeo/credentials/oidc.
Each provider also supports different grant type. Application grant types (or flows) are methods through which applications can gain an access token and by which you grant limited access to your resources to another entity without exposing credentials. Based on the need of your application, such as the level of interactivity, a different grant type could be more suitable. For use cases where processing will be done through background processes, we generally recommend using the device flow.
Most programming languages supporting HTTP communication have dedicated libraries for OIDC/OAuth2.0 authentication. Due to its complexity, we highly recommend considering using a library for this functionality.
You can always go to the openEO API documentation for more details on the authentication process and creating authorization headers based on the access token:
- [Authentication through OIDC](https://openeo.org/documentation/1.0/developers/api/reference.html#tag/Account-Management/operation/authenticate-oidc)
- [Authentication Bearer](https://openeo.org/documentation/1.0/developers/api/reference.html#section/Authentication/Bearer)
Building your openEO processing graph
OpenEO operates based on [processing graphs](https://openeo.org/documentation/1.0/glossary.html#processes) representing the workflow executed on the openEO backend by chaining processes. For example:
{
"variability_map_step": {
"process_id": "variability_map",
"arguments": {
"polygon": {
"type": "Polygon",
"coordinates": [
[
[5.1882865933798366, 51.25210584147581],
[5.187754425811659, 51.24935252933054],
[5.1927447419830015, 51.24894986862563],
[5.193456116324383, 51.25172551170871],
[5.1882865933798366, 51.25210584147581]
]
]
},
"date": ["2023-08-01T00:00:00Z", "2023-12-31T00:00:00Z"]
},
"result": true,
"namespace": "vito"
}
}The provided example illustrates the execution of the variability map service, defined at https://portal.terrascope.be/catalogue/app-details/62. This represents a basic processing graph containing a single step named variability_map_step. This step invokes the process variability_map, identified by the process_id key within the vito namespace. Input parameters such as polygon and date are specified through the arguments key. The result flag signifies that this step serves as the final one, producing the ultimate result of the processing graph.
While this example is straightforward, constructing processing graphs for more advanced workflows can be complex. Chaining different steps together and managing process arguments can be challenging. To streamline the development of advanced processing graphs, we recommend utilizing the openEO Web Editor (https://editor.openeo.org/). This online application features a graphical user interface enabling users to construct more complex processing graphs effortlessly. Users can access the full processing graph in the Code tab.
Executing a batch job
Once the openEO processing graph is ready, it can be submitted to an openEO backend to initiate the actual data processing. In this instance, our focus is on utilizing openEO batch jobs to compute variability maps for a designated time range, specified by the date parameter, and a defined area of interest, outlined by the polygon parameter. For extensive processing, especially background and time-consuming tasks, we recommend using batch jobs ([Data Processing Modes](https://openeo.org/documentation/1.0/glossary.html#data-processing-modes), [Batch Jobs](https://openeo.org/documentation/1.0/developers/api/reference.html#tag/Batch-Jobs)).
Executing a batch job on openEO involves several steps:
- Create a batch job by sending an HTTP request with vital information about the job, such as the title and processing graph ([Create Job](https://openeo.org/documentation/1.0/developers/api/reference.html#tag/Batch-Jobs/operation/create-job)). The response will contain a
Locationheader containing the URL to the newly created batch job. This URL includes an ID representing the batch job, which will be utilized in subsequent steps.
curl --location 'https://openeo.vito.be/openeo/jobs' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer [TOKEN]' \
--data '{
"title": "OpenEO API - Variability Map",
"process": {
"process_graph": {
"variability_map_step": {
"process_id": "variability_map",
"arguments": {
"polygon": {
"type": "Polygon",
"coordinates": [
[
[
5.1882865933798366,
51.25210584147581
],
[
5.187754425811659,
51.24935252933054
],
[
5.1927447419830015,
51.24894986862563
],
[
5.193456116324383,
51.25172551170871
],
[
5.1882865933798366,
51.25210584147581
]
]
]
},
"date": [
"2023-08-01T00:00:00Z",
"2023-12-31T00:00:00Z"
]
},
"result": true,
"namespace": "vito"
}
}
}
}'- Start the batch job using the job ID obtained in the previous step ([Start Job](https://openeo.org/documentation/1.0/developers/api/reference.html#tag/Batch-Jobs/operation/start-job)). Executing this request triggers openEO to schedule the execution of the batch job and provides a response once the batch job has been successfully queued.
curl --location --request POST 'https://openeo.vito.be/openeo/jobs/[BATCH JOB ID]/results' \
--header 'Authorization: Bearer [TOKEN]'- Polling the status of the batch job ([Describe Job](https://openeo.org/documentation/1.0/developers/api/reference.html#tag/Batch-Jobs/operation/describe-job)). Since batch jobs are designed for background processing, the previous request only returns a response once the batch job has been successfully queued. Therefore, the responsibility for monitoring the status of the batch job falls on the application that integrates the openEO API. The response of the polling request will include a
statusfield reflecting the latest status of the batch job.
curl --location 'https://openeo.vito.be/openeo/jobs/[BATCH JOB ID]' \
--header 'Authorization: Bearer [TOKEN]'- Downloading the batch job results after obtaining a
finishedstatus ([List Results](https://openeo.org/documentation/1.0/developers/api/reference.html#tag/Batch-Jobs/operation/list-results)). The response will provide a list of assets representing various output files generated by the batch job. Each asset includes detailed information, including anhrefproperty representing the URL to download the actual file.
curl --location 'https://openeo.vito.be/openeo/jobs/[BATCH JOB ID]/results' \
--header 'Authorization: Bearer [TOKEN]'The openEO Web Editor (https://editor.openeo.org/) serves as a valuable tool for debugging and troubleshooting any issues that may arise during the execution of openEO workflows. Upon logging in, users can access an overview of triggered batch jobs, complete with their latest status. Additionally, the platform offers the capability to view logs and download results associated with each batch job.
m-mohr
The official Python/R/JS clients support the openEO-way of OIDC based auth out of the box. But sometimes we have use cases or clients that work from a different stack, e.g. Java, C#, ... and then it takes quite a bit of explanation what has to happen practically to get to valid auth headers. It's far from trivial to figure this out directly from the openEO spec if you're new to openEO
Documentation idea: a practical, language agnostic step-by-step walk-through of what should happen to obtain valid OIDC based auth headers