Investigate catalog method for contextual data archiving

TUDelftGeodesy / stmtools

Xarray extension for Space-Time Matrix

https://tudelftgeodesy.github.io/stmtools/

Apache License 2.0

6 stars 0 forks source link

Investigate catalog method for contextual data archiving #64

Closed rogerkuou closed 5 months ago

rogerkuou commented 6 months ago

Make a workflow for cataloging contextual datasets. The purpose is to make contextual data easily to be used for enrichment by STMTools.

Download the following datasets and related metadata:

BAG (Building registrations. Represents large volume polygons, 3GB in zip file, 7GB in gpkg)
BRP bodem (Soil map. Represents small volume polygons, 140MB gpkg)
A static raster? (is this needed? AHN?)
An NetCDF file for Spatial-Temporal data from ECMWF (ERA-5 / KNMI weather station)?

Preprocessing:

CRS conversion
Digest metadata: can it be easily read in Python?

Cloud optimized saving:

What is the best file format? Should we already use chunk saving like Zarr/GeoParquet?

Create catalog
- Is the STAC catalog the best choice?

rogerkuou commented 6 months ago

Current urgency for Spatial temporal data: KNMI weather station data. @FreekvanLeijen will present a data example

rogerkuou commented 6 months ago

Request from Niels: if we can keep the json files from catelog outside the data dir; and keep in mind that the data may be moved to a new location.

rogerkuou commented 6 months ago

@FreekvanLeijen can you hare the KNMI weather station files?

FreekvanLeijen commented 6 months ago

This is the link:

https://www.daggegevens.knmi.nl/klimatologie/uurgegevens

The idea would be to insert meteo data into the STM of the data (e.g. temperature, pressure) closest to the SAR satellite acquisition. (But maybe also: the sum of rainfall in the last 3 days, etc).

Furthermore, I attached:

an example data file (just 1000 lines) knmi_stn280_example_file.txt
two old Matlab scripts, handling the data. importMeteodata_knmi.zip get_meteo.zip
.csv file with KNMI station locations knmistations.csv (see also https://github.com/MotionbyLearning/sandbox/tree/develop_vector_processing for an example implementation using these station locations made earlier (in the 'step 3' script).

Let me know in case anything is unclear.

rogerkuou commented 6 months ago

Consider using STAC extensions, e.g. STAC datacube to constraint

rogerkuou commented 5 months ago

Initial implementation in example scripts: https://github.com/MotionbyLearning/example_scripts/pull/3 More follow-up discussions in: https://github.com/TUDelftGeodesy/stmtools/discussions/70