Integration of openEO process

[santilland]

It is expected that (at least) 3 openEO processes will become available which should be integrated into GTIF. The integration concept would be similar to the one done for ship detections in the RACE instance.

The first openEO process to become available is expected to be PV farm detections. It still needs to be understood where and by who the process will run "as a service".

If EOX should take care of integrating the openEO process one possibility of on-boarding would be through the Bring Your Own Algorithm (BYOA) approach.

@jamesemwheeler will coordinate the activity with EOX as to what will become available and potential integration steps that will need to be done in the future.

*edit: potential connection to #2396

[jamesemwheeler]

https://github.com/clausmichele/openEO_photovoltaic/tree/main/udf_inference

[Patrick1G]

Should be called as an openEO UDP as below:

inference_result = conn.datacube_from_process( process_id=UDP_ID, spatial_extent={ "east": 16.414, "north": 48.008, "south": 47.962, "west": 16.342 },

[Patrick1G]

@santilland @jamesemwheeler in terms of user account and authentication, we might want to create a GTIF openEO user account, so that future UDPs (wind mill detection) can also be run accordingly. This would have to be requested from the NOR..

[santilland]

@Patrick1G after discussion with our team, as you mentioned it would make most sense to create a dedicated eodashboard / GTIF project openEO account. We will look into making a NOR request for this.

[santilland]

Hello @Patrick1G , we tried to use cdse authentication (as we discussed) for openEO and run notebooks from the repository.

We tried to use the udp notebook (openeo_pv_farms_inference_udp.ipynb) to load the process into our openEO(CDSE) account, this seems to have worked, but trying to call the process (second part of the notebook) is producing an error. We experimented also by reducing the date range (keeping same bounds as original notebook) as the call was taking over 10 minutes.

@jamesemwheeler is there a way you can integrate the UDP into an account in a way that we can directly test it?

We also experimented in using the udf notebook - as the readme says (openeo_pv_farms_inference_udf.ipynb), which also produces errors when executing it without changing the example (using same variables)

[Patrick1G]

Hi @santilland, maybe we should post this issue on the user forum, VITO colleagues are very active there and also Michele who developed the algo sees the posts: https://forum.openeo.cloud/

What could be the issue is that you have selected a quite small time period, but the UDP actually computes temporal metrics from the S2 data (min, max, median etc.). So instead of defining a 5 day time period, try it with 4-8 weeks...

[santilland]

@Patrick1G we tried the example request which has multiple weeks ["2023-05-01", "2023-09-30"] it runs for over 20 minutes. This is in any case an issue on it's own, we would need to implement handling of asynchronous jobs. But we still are getting:

Trying to construct a datacube with a bounds Extent(600075.7422801706, 5312933.126721961, 605569.3834671596, 5318171.680470274) that is not entirely inside the global bounds: Extent(600090.7400086118, 5312948.124328609, 605554.3859890398, 5318156.68286709).

Which for me sounds like a potential projection issue?

@jamesemwheeler did you run this successfully? Is my understanding correct that you have contact with the developer team? Can you follow up with them?

[clausmichele]

Hi everyone, next week I will have a look at this!

[clausmichele]

So, I've just checked and there was indeed an error in the notebook https://github.com/clausmichele/openEO_photovoltaic/blob/main/udf_inference/openeo_pv_farms_inference_udf.ipynb It was cause by this additional comma after the spatial_extent dictionary. Removing the comma solves the problem and it runs fine. I updated the notebook in the repository.

[santilland]

Hello @clausmichele thank you for your support! Removing the comma seems to let the process run further, but now we get following error.

Any idea what the issue could be?

[clausmichele]

Do you get this error running the notebook as is, cloned from the repo? Using openEO Platform or CDSE?

[santilland]

The issue happens on CDSE by running as is

[clausmichele]

Well, then it's something I can't help with. Probably it's better to ask on the forum: https://forum.dataspace.copernicus.eu/ Anyway, @HansVRP helped on the VITO side for the UDF part.

[Patrick1G]

@clausmichele Another thing to keep in mind; @santilland and colleagues are running the notebook from within EOxHub jupyterlab - this might make a difference in terms of client library versions (and other aspects?)...

[clausmichele]

Hi @Patrick1G, this really shouldn't matter, as long as the openeo library is up to date. I also tried with the same code connecting to CDSE and it fails for me too.

The code was developed within the openEO Platform project with VITO, please report this to them since they maintain both back-ends.

@santilland if you can, use openEO Platform (openeo.cloud) instead of CDSE as a workaround until they fix the problem.

[HansVRP]

as part of APEX we will be porting this UPD over in the next 2 weeks. So I will immediately take a look at where the error comes from.

[Patrick1G]

@santilland any updates on running this with an openEO platform subscription?

[Patrick1G]

as part of APEX we will be porting this UPD over in the next 2 weeks. So I will immediately take a look at where the error comes from.

@HansVRP any updates on this investigation?

[HansVRP]

currently being worked on in:

https://github.com/ESA-APEx/apex_algorithms/issues/34

[HansVRP]

Already managed to get a clean UDF running on CDSE backend. Wuld need to be validated. @clausmichele what is the standard spatiotemporal extent you used for validation

[HansVRP]

working UDF and UDP can be found in the following PR: Changes might still be made

https://github.com/ESA-APEx/apex_algorithms/pull/44

[Patrick1G]

excellent news, and FYI @santilland @jamesemwheeler

[Patrick1G]

Hi @santilland

any progress on testing the on the fly inference with openEO?? I did not see any sponsorship request, so I assume you will use the UDP in CDSE?

Lets please prioritize this issue...

[Patrick1G]

@jamesemwheeler @santilland This is the latest version of the UDP:

https://github.com/ESA-APEx/apex_algorithms/tree/main/openeo_udp/eurac_pv_farm_detection

https://github.com/ESA-APEx/apex_algorithms/blob/main/algorithm_catalog/eurac_pv_farm_detection/eurac_pv_farm_detection_description.md

[Patrick1G]

@santilland @jamesemwheeler I have a first benchmark with acceptable wall time for on the fly inference: ID: vito-j-241111e5f2b74ea8b32422d85d54984e Submitted: 11/11/2024, 2:51:17 PM UTC Updated: 11/11/2024, 2:55:50 PM UTC Image Dimensions: 132 × 213 (--> should be 2640x4260m --> 11.24km²)

Usage Metrics

• CPU usage 2,779 cpu-seconds
• Wall time 215 seconds
• Input Pixel 4.125 mega-pixel
• Memory usage 6,578,234 mb-seconds

and the process graph: { "process_graph": { "eurac_pv_farm_detection": { "arguments": { "spatial_extent": { "east": 12.971551618133702, "north": 41.77051819964615, "south": 41.759176626549845, "west": 12.946484944690127 }, "temporal_extent": [ "2023-07-01", "2023-07-31" ] }, "description": "An openEO process developed by EURAC to detect photovoltaic farms, based on sentinel2 data.", "process_id": "eurac_pv_farm_detection", "result": true } } }