configure and test INCD layer in openEO

[jdries]

https://resto.c-scale.zcu.cz/collections/S2/items

[jdries]

Copy SENTINEL_L2A collection from: https://github.com/Open-EO/openeo-geotrellis-kubernetes/blob/master/docker/creo_layercatalog.json

[jdries]

There's now also a second collection, by CESNet: https://resto.c-scale.zcu.cz/collections/S2-experimental/items

[JeroenVerstraelen]

Support for S2 and S2-experimental features has been added to gwc-geotrellis (develop branch) + added tests. Now working on creating the openeo layer for S2: SENTINEL2_L1C_INCD

Note: the S2 layer currently only contains 2 features.

[JeroenVerstraelen]

Current version of the layer in layercatalog.json: https://gist.github.com/JeroenVerstraelen/4450069fd217b8088791b4c6aa25e582

Any feedback on this layer definition is welcome.

[jdries]

Some feedback:

• adjust band names to match asset names and the recommended names for Sentinel-2: https://docs.openeo.cloud/federation/backends/collections.html#sentinel2-l2a so this is B01,B02... (now you have B1,B2,...)
• in the description, mention that this layer is experimental and for C-Scale project only, also add 'experimental' keyword
• add cube:dimensions property, like in layer for creo, this is important to retrieve the properties of the datacube, band names again have to match

[jdries]

The basic data access now seems to work when we test it directly. There's only a weird gdal issue that happens when reading with multiple threads, while this works fine on the creo jpeg2000 files. Temporary workaround could just be to avoid this parallelism. Next step is to expose the incd layer on terrascope instance, for further testing.

[jdries]

The INCD layer on Terrascope now seems to produce first data, next step is to provide layer config into INCD setup.

[zbenta]

@jdries any thoughts on how to create a layercatalog.json to expose the data we registered on cesnet on our openeo endpoint?

[jdries]

Hi @zbenta, the json file is already there: https://github.com/Open-EO/openeo-geotrellis-kubernetes/blob/master/docker/cscale_layercatalog.json Now you only need to configure it in your instance. At INFN @maricaantonacci already figured out how to do this: https://openeo.vm.fedcloud.eu/openeo/1.1.0/collections/SENTINEL2_L1C_INCD

[zbenta]

@jdries I had already done that: https://openeo.a.incd.pt/openeo/1.1.0/collections/SENTINEL2_L1C_INCD But as you can see the catalog only states that we have data from 27-12-2020 to 30-12-2020

How can I copy the data from cesnet: https://resto.c-scale.zcu.cz/collections/S2 So I can get all the data we registered there, from 02-01.2028 to 30-12-2020?

[jdries]

Actually this is for now just hardcoded in the json, so you can simply adjust it in there. In fact, this is purely informative for the user, so actual processing should be able to work with all the data that can be found in the collection.

[jdries]

@zbenta I tested your backend again, using this simple script that avoids requiring batch jobs and thus zookeeper:

con = openeo.connect("https://openeo.a.incd.pt/openeo/1.1.0/").authenticate_oidc()
    cube = (
        con.load_collection(
            "SENTINEL2_L1C_INCD",
            temporal_extent=["2020-12-27", "2020-12-28"],
            bands=[ "B02"],
            spatial_extent={"west": -5.0, "south": 37.0, "east": -4.0, "north": 38.0},
        )
    )

cube.download("incd-test-endpoint.tiff")

This now no longer works, because gdal is unable to open the file. I had a look at the STAC catalog and noticed the returned urls have changed to s3 urls like this: s3://sentinel-s2-l1c/tiles/29/S/PB/2020/12/27/0/B05.jp2 This is the most likely reason why it does not work.

Direct S3 access can work, but we need to tell gdal about some parameters, these are the most relevant environment variables that need to be set correctly in your values.yaml, can you check:

    AWS_DIRECT: "TRUE"
    AWS_ACCESS_KEY_ID: "XXX"
    AWS_SECRET_ACCESS_KEY: "YYY"
    AWS_S3_ENDPOINT: "data.incd.com"
    AWS_DEFAULT_REGION: "RegionOne"
    AWS_REGION: "RegionOne"
    AWS_HTTPS: "NO"
    AWS_VIRTUAL_HOSTING: "FALSE"
    CPL_DEBUG: "ON"

[zbenta]

Hi @jdries thanks for the input

We have to look into our metadata manipulation script and figure out what went wrong, because we have code to set the correct url in the metadata

  #images
    imagelist=getMTD(manifesturl,"dataObject","","IMG_DATA_Band_")
  #loop through existing images
    k=0
    for i in range(1,13):
      imagename=""
      if i<10:
        imagename="B0"+str(i)
      else:
        imagename="B"+str(i)
    #print the images
    #print("IMAGENAME:",imagename)
    #print("URL:",dataurl+imagelist[k][22:-21])
      jsonitem['assets'][imagename]["href"]=dataurl+imagelist[k][22:-21]
      k+=1
  #print the resulting transformation
  #data=jsonitem['assets']["B01"]["href"]
  #print("the image data:",data)

But looking into the catalog we see the error you report, but the thumbnail has the correct link into our s3 bucket

[zbenta]

We have updated the resto catalog, thanks @jdries for the heads up. We also tried your code again and got a timeout:

Failed to parse API error response: 504 '<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n'

---------------------------------------------------------------------------
OpenEoApiError                            Traceback (most recent call last)
/tmp/ipykernel_242/439246359.py in <module>
      3 cube = ( con.load_collection("SENTINEL2_L1C_INCD",temporal_extent=["2020-12-27T00:00:00Z", "2020-12-28T00:00:00Z"],bands=[ "B02"],spatial_extent={"west": -7.8897490816, "south": 36.0074603289, "east": -4.0014631782, "north": 38.8443648229},))
      4 
----> 5 cube.download("incd-test-endpoint.tiff")
      6 
      7 """

/opt/conda/lib/python3.9/site-packages/openeo/rest/datacube.py in download(self, outputfile, format, options)
   1611         # TODO: only add `save_result` node when there is none yet?
   1612         cube = self.save_result(format=format, options=options)
-> 1613         return self._connection.download(cube.flat_graph(), outputfile)
   1614 
   1615     def validate(self) -> List[dict]:

/opt/conda/lib/python3.9/site-packages/openeo/rest/connection.py in download(self, graph, outputfile, timeout)
   1098         """
   1099         request = self._build_request_with_process_graph(process_graph=graph)
-> 1100         response = self.post(path="/result", json=request, expected_status=200, stream=True, timeout=timeout)
   1101 
   1102         if outputfile is not None:

/opt/conda/lib/python3.9/site-packages/openeo/rest/connection.py in post(self, path, json, **kwargs)
    171         :return: response: Response
    172         """
--> 173         return self.request("post", path=path, json=json, allow_redirects=False, **kwargs)
    174 
    175     def delete(self, path, **kwargs) -> Response:

/opt/conda/lib/python3.9/site-packages/openeo/rest/connection.py in request(self, method, path, headers, auth, check_error, expected_status, **kwargs)
    596         try:
    597             # Initial request attempt
--> 598             return _request()
    599         except OpenEoApiError as api_exc:
    600             if api_exc.http_status_code == 403 and api_exc.code == "TokenInvalid":

/opt/conda/lib/python3.9/site-packages/openeo/rest/connection.py in _request()
    589         # Do request, but with retry when access token has expired and refresh token is available.
    590         def _request():
--> 591             return super(Connection, self).request(
    592                 method=method, path=path, headers=headers, auth=auth,
    593                 check_error=check_error, expected_status=expected_status, **kwargs,

/opt/conda/lib/python3.9/site-packages/openeo/rest/connection.py in request(self, method, path, headers, auth, check_error, expected_status, **kwargs)
    119         expected_status = ensure_list(expected_status) if expected_status else []
    120         if check_error and status >= 400 and status not in expected_status:
--> 121             self._raise_api_error(resp)
    122         if expected_status and status not in expected_status:
    123             raise OpenEoRestError("Got status code {s!r} for `{m} {p}` (expected {e!r})".format(

/opt/conda/lib/python3.9/site-packages/openeo/rest/connection.py in _raise_api_error(self, response)
    150             else:
    151                 exception = OpenEoApiError(http_status_code=status_code, message=text)
--> 152         raise exception
    153 
    154     def get(self, path, stream=False, auth: AuthBase = None, **kwargs) -> Response:

OpenEoApiError: [504] unknown: <html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx</center>
</body>
</html>

[jdries]

This is most likely caused by the layercatalog.json file which uses the default rather than your custom version. What did we use again to fix that in the main webapp, a config map?

[zbenta]

This is most likely caused by the layercatalog.json file which uses the default rather than your custom version. What did we use again to fix that in the main webapp, a config map?

Exactly, we did a configmap.

[jdries]

And you probably had to set OPENEO_CATALOG_FILES as well, what is the custom location? I can probably improve the batch job script to use the custom location as well.

[zbenta]

And you probably had to set OPENEO_CATALOG_FILES as well, what is the custom location? I can probably improve the batch job script to use the custom location as well.

OPENEO_CATALOG_FILES: "/opt/layercatalog.json"

[jdries]

@tcassaert we would need to use the contents of the driver layercatalog file when we create a new batch job, instead of pointing to the default layercatalog in the container. Is there some way to do that? Should we also create a new configmap for every batch job containing those layercatalog files?

Or should we start to consider that we make the whole batch job template configurable: https://github.com/Open-EO/openeo-geopyspark-driver/blob/master/openeogeotrellis/deploy/sparkapplication.yaml.j2 Otherwise we may continue bumping into one issue after the other, until this whole file is somehow dynamic anyway.

[Jaapel]

Not sure what the status is since the last update here,

I just tried the INCD Layer via the VITO backend (collection id SENTINEL2_L1C_INCD) and this works just fine when loading data and applying some filter (I cannot continue with just 3 days of data). I can either look into a larger area and comparing speed with VITO's collection, or go with the new endpoint and see what is available. For now I go with the second 👍 .

[zbenta]

Hi guys, any idea how can we generate a layercatalog.json file from a given set of products? Is there any tool available to do so? We just copied the file @jdries suggested, but we would like to make available other collections and are not aware of how to go about generating the file and using it to create a new collection in our resto catalog.

Hi @zbenta, the json file is already there: https://github.com/Open-EO/openeo-geotrellis-kubernetes/blob/master/docker/cscale_layercatalog.json Now you only need to configure it in your instance. At INFN @maricaantonacci already figured out how to do this: https://openeo.vm.fedcloud.eu/openeo/1.1.0/collections/SENTINEL2_L1C_INCD

[jdries]

Hi @zbenta , there is no such tool. Basically you need to create a valid STAC collection in resto yourself, and then copy paste an entry in layercatalog.json. openEO will try to reuse metadata from the stac collection, so you can keep openEO config to a minimum by putting all the metadata in STAC.

[zbenta]

Thanks for the feedback @jdries, well try to build our own file and create a new collection with the new data we have collected. I'll keep you guys posted.

[zbenta]

Hi guys, we are having issues while trying to authenticate into our endpoint. We have no identity providers whenever we try using the notebook @Jaapel developed. We confirm, it if we go to the endpoint where the list of identity providers would appear: https://openeo.a.incd.pt/openeo/1.1.0/credentials/oidc

Any ideas on how to go about and solve this issue?

[soxofaan]

FYI: since April 2023 (https://github.com/Open-EO/openeo-python-driver/commit/d46daf6eb57466f3ae1b986abfa9ea25a7ce22c4) openeo-python-druver does not have a hardcoded OIDC providers list anymore, and this has to be provided for a given deployment through the config system. I guess this openeo.a.incd.pt endpoint depends on the old hardcoded OIDC provider list. Assuming this deployment is a openeo-geopyspark-driver deployment, there is some documentation on this configuration aspect at https://github.com/Open-EO/openeo-geopyspark-driver/blob/master/docs/configuration.md

I hope this helps

[zbenta]

We added the changes as suggested per @soxofaan , but we now have a new issue, we can authenticate with "egi", but we are unable to do so with "google". We defined our identity providers as follows:

from openeo_driver.users.oidc import OidcProvider
    from openeogeotrellis.config import GpsBackendConfig
    config = GpsBackendConfig(
        oidc_providers=[
          OidcProvider(id="egi", title="EGI Check-in", issuer="https://aai.egi.eu/auth/realms/egi/", scopes=["openid","email","eduperson_entitlement","eduperson_scoped_affiliation"], default_clients=[{"grant_types": ["authorization_code+pkce","urn:ietf:params:oauth:grant-type:device_code+pkce","refresh_token"],"id": "vito-default-client","redirect_urls": ["https://editor.openeo.org","http://localhost:1410/"]}]),
          OidcProvider(id="google", title="Google", issuer="https://accounts.google.com/", scopes=["max"], default_clients=[{"grant_types": ["authorization_code","urn:ietf:params:oauth:grant-type:device_code","refresh_token"],"id": "google","redirect_urls": ["https://editor.openeo.org","http://localhost:1410/"]}]),
        ],
    )

Are we missing something in the "google" provider?

Can we create a local account for authentication? Where can we find documentation for that?

[soxofaan]

default_clients=[{ ... "id": "google"

it has been a while I've setup a OIDC client in google, but "google" does not look ike a valid client id. Note that you can not choose that id yourself, you have to go through some kind of registration procedure in google accounts to obtain such an id. I've also never seen scopes=["max"] is that a documented feature of google OIDC?

Is there a particular reason you want to set up google as a OIDC provider for your backend? I think google is one of the possible options in EGI check-in, so setting up a dedicated google OIDC provider on your openeo backend seems overkill then?

[zbenta]

We need to grant access to the platform to an external entity, that hasn't got a egi account, so we were wondering if it was possible to use google as an authenticator for an external person. By the way, can we create a local account? The documentation doesn't seem to be very clear on that topic.

"And now, for something completely different." We are having issues submitting jobs, we are using a jupyter notebook to interact with the endpoint. We send a job and get the following result:

We looked to the spark-driver pod the logs regarding this specific job(j-240125d633aa4254ac2d9ed88ee73f77), and what jumps out is the following line:

{"message": "Exception when calling CustomObjectsApi->list_custom_object: (422)\nReason: Unprocessable Entity\nHTTP response headers: HTTPHeaderDict({'Audit-Id': 'bceceadf-5f3c-4dbf-b940-fd86bc3c09a2', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': 'bd549fbc-ebe5-4c1b-a000-37e49b58c332', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'ff4f3b61-23c8-4a29-a892-3b6f791730b5', 'Date': 'Thu, 25 Jan 2024 09:16:27 GMT', 'Content-Length': '1629'})\nHTTP response body: {\n  \"kind\": \"Status\",\n  \"apiVersion\": \"v1\",\n  \"metadata\": {},\n  \"status\": \"Failure\",\n  \"message\": \"SparkApplication.sparkoperator.k8s.io \\\"a-e341aa248e9b4433841302123f961b2d\\\" is invalid: [metadata.labels: Invalid value: \\\"98dfa532579388c49bea11b50d390929551d2990ac826d5ef49a602ee7c60d97@egi.eu\\\": must be no more than 63 characters, metadata.labels: Invalid value: \\\"98dfa532579388c49bea11b50d390929551d2990ac826d5ef49a602ee7c60d97@egi.eu\\\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')]\",\n  \"reason\": \"Invalid\",\n  \"details\": {\n    \"name\": \"a-e341aa248e9b4433841302123f961b2d\",\n    \"group\": \"sparkoperator.k8s.io\",\n    \"kind\": \"SparkApplication\",\n    \"causes\": [\n      {\n        \"reason\": \"FieldValueInvalid\",\n        \"message\": \"Invalid value: \\\"98dfa532579388c49bea11b50d390929551d2990ac826d5ef49a602ee7c60d97@egi.eu\\\": must be no more than 63 characters\",\n        \"field\": \"metadata.labels\"\n      },\n      {\n        \"reason\": \"FieldValueInvalid\",\n        \"message\": \"Invalid value: \\\"98dfa532579388c49bea11b50d390929551d2990ac826d5ef49a602ee7c60d97@egi.eu\\\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')\",\n        \"field\": \"metadata.labels\"\n      }\n    ]\n  },\n  \"code\": 422\n}\n\n", "levelname": "ERROR", "name": "openeogeotrellis.backend", "created": 1706174224.3515952, "filename": "backend.py", "lineno": 2356, "process": 138, "job_id": "j-240125d633aa4254ac2d9ed88ee73f77", "user_id": "98dfa532579388c49bea11b50d390929551d2990ac826d5ef49a602ee7c60d97@egi.eu", "req_id": "r-24012564be494238b902db37b3a2786f"}

Any thoughts on this one?

[sebastian-luna-valero]

Just my two cents.

We need to grant access to the platform to an external entity, that hasn't got a egi account, so we were wondering if it was possible to use google as an authenticator for an external person.

The external person could use his/her google account to create an EGI account (for free): https://docs.egi.eu/users/aai/check-in/signup/

Then, the access problem should be resolved.

I hope that helps!

[cesarpferreira]

We need to grant access to the platform to an external entity, that hasn't got a egi account, so we were wondering if it was possible to use google as an authenticator for an external person. By the way, can we create a local account? The documentation doesn't seem to be very clear on that topic.

"And now, for something completely different." We are having issues submitting jobs, we are using a jupyter notebook to interact with the endpoint. We send a job and get the following result:

We looked to the spark-driver pod the logs regarding this specific job(j-240125d633aa4254ac2d9ed88ee73f77), and what jumps out is the following line:

{"message": "Exception when calling CustomObjectsApi->list_custom_object: (422)\nReason: Unprocessable Entity\nHTTP response headers: HTTPHeaderDict({'Audit-Id': 'bceceadf-5f3c-4dbf-b940-fd86bc3c09a2', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': 'bd549fbc-ebe5-4c1b-a000-37e49b58c332', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'ff4f3b61-23c8-4a29-a892-3b6f791730b5', 'Date': 'Thu, 25 Jan 2024 09:16:27 GMT', 'Content-Length': '1629'})\nHTTP response body: {\n  \"kind\": \"Status\",\n  \"apiVersion\": \"v1\",\n  \"metadata\": {},\n  \"status\": \"Failure\",\n  \"message\": \"SparkApplication.sparkoperator.k8s.io \\\"a-e341aa248e9b4433841302123f961b2d\\\" is invalid: [metadata.labels: Invalid value: \\\"98dfa532579388c49bea11b50d390929551d2990ac826d5ef49a602ee7c60d97@egi.eu\\\": must be no more than 63 characters, metadata.labels: Invalid value: \\\"98dfa532579388c49bea11b50d390929551d2990ac826d5ef49a602ee7c60d97@egi.eu\\\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')]\",\n  \"reason\": \"Invalid\",\n  \"details\": {\n    \"name\": \"a-e341aa248e9b4433841302123f961b2d\",\n    \"group\": \"sparkoperator.k8s.io\",\n    \"kind\": \"SparkApplication\",\n    \"causes\": [\n      {\n        \"reason\": \"FieldValueInvalid\",\n        \"message\": \"Invalid value: \\\"98dfa532579388c49bea11b50d390929551d2990ac826d5ef49a602ee7c60d97@egi.eu\\\": must be no more than 63 characters\",\n        \"field\": \"metadata.labels\"\n      },\n      {\n        \"reason\": \"FieldValueInvalid\",\n        \"message\": \"Invalid value: \\\"98dfa532579388c49bea11b50d390929551d2990ac826d5ef49a602ee7c60d97@egi.eu\\\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')\",\n        \"field\": \"metadata.labels\"\n      }\n    ]\n  },\n  \"code\": 422\n}\n\n", "levelname": "ERROR", "name": "openeogeotrellis.backend", "created": 1706174224.3515952, "filename": "backend.py", "lineno": 2356, "process": 138, "job_id": "j-240125d633aa4254ac2d9ed88ee73f77", "user_id": "98dfa532579388c49bea11b50d390929551d2990ac826d5ef49a602ee7c60d97@egi.eu", "req_id": "r-24012564be494238b902db37b3a2786f"}

Any thoughts on this one?

Also, we had to comment local:///opt/geotrellis-backend-assembly-static.jar from jarDependencies in our values file because the file was missing in /opt directory inside the myspark-driver pod with the image latest tag. We are still using version 0.7.2 for the sparkapplication helm chart and the spark-operator chart is already on the latest version (1.1.27). Could our problems be related to that missing file? Or maybe an incompatibility problem between the image version (latest), the sparkapplication version (0.7.2) and the spark-operator version (1.1.27)?

We also tried to deploy sparkapplication chart on version 0.14.5 but we got errors about missing settings in our values:

helm install myspark sparkapp/sparkapplication --namespace spark-jobs -f values.yaml
Error: INSTALLATION FAILED: template: sparkapplication/templates/sparkapplication_ha.yaml:1:14: executing "sparkapplication/templates/sparkapplication_ha.yaml" at <.Values.ha.enabled>: nil pointer evaluating interface {}.enabled

These settings are not talked about in the chart README.

Any ideas on how we could upgrade our chart to the latest version with a correct values file?