Implement Job splitter and manager in GFMAP

Open-EO / openeo-gfmap

Generic framework for EO mapping applications building on openEO

Apache License 2.0

4 stars 0 forks source link

Implement Job splitter and manager in GFMAP #12

Closed GriffinBabe closed 3 months ago

GriffinBabe commented 5 months ago

Implement a job splitter and a job manager in OpenEO GFMAP:

The Job Splitter takes in parameter a dataset containing polygons/points extraction location and dates. It first separates the job by the temporal-extent and then by the spatial-extent (by default, the spatial split is performed using the s2-grid, to avoid launching jobs that are too large on OpenEO)
The Job Manager launches jobs and keep track of their progress. If a job fails, logs are saved to a location and the user will be notified of the location of those logs. If the job is successful, the products are downloaded and a post-job script is run locally (if the user specifies it).

Maybe, if too many OpenEO jobs fail then cancel all the other jobs and abort the task.

kvantricht commented 5 months ago

@VincentVerelst you can find an example of dataset chunking here.

kvantricht commented 4 months ago

Resolution 4 in https://h3geo.org/docs/core-library/restable/
Crucial to make sure the splitter takes into account the UTM zones.

GriffinBabe commented 4 months ago

@kvantricht From the dataset provided in issue #40 here is a first result from using the h3 library. Seems pretty good to me except maybe for the hexagons including a few points

If I select only the points with extract=True field

GriffinBabe commented 4 months ago

@soxofaan I would also be interested in your opinion on this method as you might be more experienced with the technical limitations of OpenEO. Each hexagon here has an approximative 30km x 30km diameter

GriffinBabe commented 4 months ago

@kvantricht can I use the dataset you provided me as a unit test resource?

kvantricht commented 4 months ago

Looks pretty good indeed. I don't think the ones with just few points are a real problem at the moment? Might be a bit of overhead but seems not a priority to address that, do you think?

kvantricht commented 4 months ago

@kvantricht can I use the dataset you provided me as a unit test resource?

in principle yes but isn't it a bit big for that?

soxofaan commented 4 months ago

@soxofaan I would also be interested in your opinion on this method as you might be more experienced with the technical limitations of OpenEO. Each hexagon here has an approximative 30km x 30km diameter

That looks fine, I don't expect a problem here

GriffinBabe commented 4 months ago

Good For the unit tests I will upload a subsampled version of it @kvantricht

GriffinBabe commented 3 months ago

Implemented and about to be merged #45