Generic implementations for load_collection and save_result

[LukeWeidenwalker]

This came out of a private chat with @clausmichele, we were talking about why load_collection isn't currently implemented in this repo.

The reason is basically this:

• load_collection

  • The id parameter in load_collection needs to be statically resolved within the backend. Previously this got resolved in openeo-odc in a terrible way. Now in our backend, we define a custom load_collection function (with a EODC_LOCAL_STAC_MAP that maps collection_id to where we store it) and register it to the process_registry later. This doesn't belong into this repo, because it's backend specific and I really want to keep this repo free of configuration.
  • If load_collection would take a STAC collection instead, this wouldn't be a problem and we could just build a generic way to load xarrays from STAC collections. This would make our lives easier, so we'll probably start discussing this at https://github.com/Open-EO/openeo-processes/issues/377
  • For the time being, I think we could still build load_stac_collection in openeo-processes-dask, and define custom load_collections to do this mapping wherever the process graph is actually executed (our backend, clientside processing).

• save_result

  • An implementation of this needs to know the destination where to save results to, therefore the implementation will depend on how the backend implements the user-workspace. This might be a filepath on a shared filesystem, an S3 bucket or something else. However, the save_result process doesn't specify this information, so this is configuration that should be added wherever the process graph is actually executed, not in a repo with generic process implementations!

For these reasons I really don't see how we'd be able to write a generic implementation for either load_collection or save_result at the moment, and that is why they aren't currently included in this repo.

A simple workaround is to do the following wherever you execute the process (before obviously!):

def load_collection(*args, **kwargs):
    ...

def save_result(*args, **kwargs):
    ...

process_registry["load_collection"] = load_collection
process_registry["save_result"] = save_result

And define what these processes should do there.

[LukeWeidenwalker]

@SerRichard

[clausmichele]

Thanks for the clarification. Documenting how to implement a base load_collection and save_result could be already a good start! I will create a PR for this, since it is anyway needed the client side processing.

[clausmichele]

@LukeWeidenwalker opendatacube (datacube) is a dependency of this project. It is used in aggregate_spatial, which supposes that the data comes from opendatacube (for attribute geobox).

How do we address this? Having a different implementation of load_collection, not based on ODC, would make this process not usable.

Edit: similar issue with resample_cube_spatial which relies on the geobox attribute.

[LukeWeidenwalker]

@LukeWeidenwalker opendatacube (datacube) is a dependency of this project. It is used in aggregate_spatial, which supposes that the data comes from opendatacube (for attribute geobox).

How do we address this? Having a different implementation of load_collection, not based on ODC, would make this process not usable.

Edit: similar issue with resample_cube_spatial which relies on the geobox attribute.

Ah, that is a very good point, I hadn't realised that connection before!

I think we could use https://github.com/opendatacube/odc-geo explicitly to make sure that this geobox object exists on all datacubes (possibly enforcing this with _normalise_output_datacube). Bit hesitant because aggregate_spatial might yet change to not need this after all, but at least for resample_cube_spatial I've already done some shopping around other libraries and haven't found anything that achieves this spatial resampling in a way that is dask-friendly. What do you think?

[clausmichele]

Sample implementations are mentioned in the docs here: https://openeo.org/documentation/1.0/developers/backends/xarray.html#the-load-collection-and-save-result-process