12 job splitters - Githubissues

Open-EO / openeo-gfmap

Generic framework for EO mapping applications building on openEO

Apache License 2.0

4 stars 0 forks source link

12 job splitters #45

Closed GriffinBabe closed 3 months ago

GriffinBabe commented 4 months ago

Closes #11 and #12

Added job splitter using h3 library. Adding the h3 library and using it directly is more lightweight in terms of package and drastically more computationally efficient than having a pre-generated dataset containing the h3 grid. Especially when it comes to low resolutions.
Added job manager, extending MultiBackendJobManager. The job manager also implements post-job actions that are computed on other threads to keep the status update smooth.

TODO before merging:

[x] Polish and add a demo notebook to the branch

GriffinBabe commented 4 months ago

@soxofaan I'm confused, there is a lint check that fails on black but the black check is disabled in the workflow file? I also tried fixing the files with black but doesn't help

GriffinBabe commented 4 months ago

@soxofaan Do you think I should be using multiprocessing instead of multithreading? In case GFMAP is running on a VM with multiple CPUs that would really accelerate possibly heavy post-job actions

soxofaan commented 4 months ago

@soxofaan I'm confused, there is a lint check that fails on black but the black check is disabled in the workflow file?

That's indeed weird. What might be the case is Gihub runs the workflow from main (instead of your branch) because you are making a pull request targeting main.

soxofaan commented 4 months ago

Do you think I should be using multiprocessing instead of multithreading? In case GFMAP is running on a VM with multiple CPUs that would really accelerate possibly heavy post-job actions

If I remember correctly, threading and multiprocessing have a common API, so I guess it's most future proof to stay within this common API, so that you can switch when one approach doesn't cut it

GriffinBabe commented 4 months ago

@kvantricht also corresponds to #43

GriffinBabe commented 3 months ago

Problem remaining with the "to_scl_dilation_mask", except that everything should be ready for merge

kvantricht commented 3 months ago

Problem remaining with the "to_scl_dilation_mask", except that everything should be ready for merge

is that something that needs to be fixed before merging or do we create another issue for that? What exactly was the problem?

GriffinBabe commented 3 months ago

Created issue for to_scl_dilation_mask and for the STAC AssetDefinition