search

Open-EO / openeo-geopyspark-driver

OpenEO driver for GeoPySpark (Geotrellis)
Apache License 2.0
25 stars 4 forks source link

[EPIC] On-premise/HPC execution mode #761

Open jdries opened 2 months ago

jdries commented 2 months ago

Project: EOEPCA Tentative deadline: end of 2024 Epic: EOEPCA/roadmap#39

Background: multiple projects and users want to run openEO on-premise. Sometimes for local debugging, but also because they have their own processing clusters. Large initiatives like Destination Earth also target HPC infra. It makes a lot of sense, because these resources are available in Europe.

Proposed general solution: We already tried to run a spark cluster on HPC, which actually works. It is however very likely to be problematic for an inexperienced operator to run, because of the inherent complexity. When considering that HPC nodes can be rather large, I propose to also target running openEO jobs on single machines, in local mode. This then solves multiple problems at once, and reduces the complexity. We would also not require many auxiliary components.

Note that I propose to simply run batch jobs, and not run the full openEO web services. Even though that would also be possible, it would again be more complex...

Subtasks (to be extended):

jdries commented 2 months ago

@JeroenVerstraelen I assigned this epic to you, to consider in longer term planning, not for you to do all the work!