Perform preprocessing to geoparquet from patch extractions

WorldCereal / worldcereal-classification

This repository contains the classification module of the WorldCereal system.

https://esa-worldcereal.org/

MIT License

19 stars 2 forks source link

Perform preprocessing to geoparquet from patch extractions #101

Open kvantricht opened 2 months ago

kvantricht commented 2 months ago

Starting from patch extractions, we need a workflow that performs:

[x] loading extractions as openEO datacubes
[x] perform preprocessing
[x] perform pixel sampling
[x] export result as geoparquet

VincentVerelst commented 2 months ago

Currently it's not possible on the CDSE backend to load_stac multiple CRS: https://github.com/Open-EO/openeo-geopyspark-driver/issues/827

VincentVerelst commented 1 month ago

Implemented a STAC splitter in gfmap to circumvent the load_stac issue: https://github.com/Open-EO/openeo-gfmap/pull/146

kvantricht commented 1 month ago

Needs #125 first.

jdries commented 3 weeks ago

moving to public mount is possible, but this would trigger a huge update in stac metadata

VincentVerelst commented 2 weeks ago

Successfully loaded precomposited METEO from STAC on Terrascope; this gives a huge performance boost
Was able to generate a parquet file with preprocessed data of meteo, dem, s1 and s2 timeseries, using some random points
/vitodata/worldcereal_data should be available on openeo-dev.vito.be soon (waiting on integration tests). This would mean we can load METEO and the Sentinel 1 and 2 patches from STAC from there
For interaction with the RDM API we are still dependent on two issues, see: https://github.com/WorldCereal/worldcereal-classification/issues/125

kvantricht commented 2 weeks ago

That's great progress Vincent, thanks! So once the RDM interaction is there we have in principle everything in place to start gradually building this geoparquet and performing some downstream tests on them?

VincentVerelst commented 2 weeks ago

That's great progress Vincent, thanks! So once the RDM interaction is there we have in principle everything in place to start gradually building this geoparquet and performing some downstream tests on them?

Indeed!