Open-EO / openeo-geopyspark-driver

OpenEO driver for GeoPySpark (Geotrellis)
Apache License 2.0
26 stars 4 forks source link

Correct scale and offset for more layers #831

Closed EmileSonneveld closed 2 months ago

EmileSonneveld commented 3 months ago

As Kasper Bonte said, there are quite some layers that don't specify the scale/offset they need. It we could read the correct offset/scale from the tiff files to put them all right at once. And make it an auto test

(Related: https://github.com/Open-EO/openeo-geopyspark-driver/issues/338 )

EmileSonneveld commented 3 months ago

Made commit in openeo-deploy: "Fix scale and offset for TERRASCOPE_S2_FAPAR_V2."

EmileSonneveld commented 3 months ago

Made a MR in openeo-deploy that fixes the offset and scale for more layers. I also moved some properties from eo:bands to raster:bands. To double check the changes I made a script that summarizes what changed:

compare_catalogs.py ```python import json with open(".../layercatalog_before.json") as f: layercatalog_before = json.load(f) with open(".../openeo-deploy/mep/layercatalog.json") as f: layercatalog_after = json.load(f) def extract_info(layercatalog): summary = set() for collection in layercatalog: if "summaries" in collection: if "eo:bands" in collection["summaries"]: eo_bands = collection["summaries"]["eo:bands"] for band_item in eo_bands: for key, value in band_item.items(): summary.add(f"""{collection["id"]} {key} {value}""") if "raster:bands" in collection["summaries"]: raster_bands = collection["summaries"]["raster:bands"] for band_item in raster_bands: for key, value in band_item.items(): summary.add(f"""{collection["id"]} {key} {value}""") return summary before = extract_info(layercatalog_before) after = extract_info(layercatalog_after) print() print("==== Unique to before change =============================") print("\n".join(before.difference(after))) print("==== Unique to after change =============================") print("\n".join(after.difference(before))) print("=================================") ```

Output:

==== Unique to before change =============================
CGLS_LAI_V2_GLOBAL statistics {'minimum': 0, 'maximum': '210'}
==== Unique to after change =============================
TERRASCOPE_S2_FCOVER_V2 offset 0
S1_GRD_SIGMA0_ASCENDING offset 29
CGLS_LAI_V2_GLOBAL statistics {'minimum': 0, 'maximum': 210}
SENTINEL1_GRD_SIGMA0 offset 29
S1_GRD_SIGMA0_DESCENDING offset 29
TERRASCOPE_S2_RHOW_V1 scale 0.01
S1_GRD_SIGMA0_DESCENDING scale 0.0005
TERRASCOPE_S2_FCOVER_V2 scale 0.005
SENTINEL1_GRD_SIGMA0 scale 0.0005
S1_GRD_SIGMA0_ASCENDING scale 0.0005
=================================
EmileSonneveld commented 3 months ago

Can't visualize the changes on editor.openeo.org https://editor.openeo.org/?server=https%3A%2F%2Fopeneo-dev.vito.be%2Fopeneo%2F Electro-Optical and Raster Imagery sections are not displayed. Also for http://openeo.vito.be Would be nice to check with an editor that does show them

EmileSonneveld commented 3 months ago

Is ok when checking catalog JSON files. For example: https://openeo-dev.vito.be/openeo/1.2/collections/S1_GRD_SIGMA0_ASCENDING

EmileSonneveld commented 2 months ago

Looks fine on openeo.vito.be. Example: image