fix catalog config for utm based layers with incomplete catalog info

[jdries]

Some catalogs (oscars) do not advertise the native crs of a product for utm based collections. I added a flag to our catalogclient to enable an automatic lookup of crs based on extent center point.

We now need to check if all our collections set this flag correctly, because we now determine it based on target resolution, which is not accurate.

This issue depends on: https://github.com/Open-EO/openeo-geopyspark-driver/issues/219 where we will refactor creation of opensearch client

[jdries]

The relevant line in Sentinel2PyramidFactory: OpenSearchClient(openSearchEndpointUrl, isUTM = maxSpatialResolution == CellSize(10.0,10.0))

Layers to check: Sentinel-5p WorldCover Copernicus DEM

[jdries]

for coherence, the flag should be set to false, because we do not manage to correctly determine crs.

[JeroenVerstraelen]

I checked all layers with (10,10) cellsize in layercatalog.json with gdalinfo:	line_number	collection_id	file_type	projection	comments
344	SENTINEL1_GRD_SIGMA0	file-s2	PROJCRS["WGS 84 / UTM zone 39N"
436	S1_GRD_SIGMA0_ASCENDING	file-s2	PROJCRS["WGS 84 / UTM zone 39N"
779	TERRASCOPE_S2_FAPAR_V2	file-s2	PROJCRS["WGS 84 / UTM zone 40S"
856	TERRASCOPE_S2_NDVI_V2	file-s2	PROJCRS["WGS 84 / UTM zone 40S"
938	TERRASCOPE_S2_LAI_V2	file-s2	PROJCRS["WGS 84 / UTM zone 40S"
1017	TERRASCOPE_S2_FCOVER_V2	file-s2	PROJCRS["WGS 84 / UTM zone 40S"
1254	TERRASCOPE_S2_TOC_V2	file-s2	PROJCRS["WGS 84 / UTM zone 40S"
1337	TERRASCOPE_S1_SLC_COHERENCE_V1			COHERENCE in name
1386	SENTINEL1_GAMMA0_SENTINELHUB	sentinel-hub
1407	SENTINEL1_GRD	sentinel-hub
1631	SENTINEL2_L1C_INCD	file-s2	PROJCRS["WGS 84 / UTM zone 29N"	https://resto.c-scale.zcu.cz
1804	SENTINEL2_L2A_SENTINELHUB	sentinel-hub
3269	TERRASCOPE_S2_RHOW_V1	file-s2	PROJCRS["WGS 84 / UTM zone 31N"
3340	TERRASCOPE_S2_CHL_V1	file-s2	PROJCRS["WGS 84 / UTM zone 31N"
3407	TERRASCOPE_S2_SPM_V1	file-s2	PROJCRS["WGS 84 / UTM zone 31N"
3475	TERRASCOPE_S2_TUR_V1	file-s2	PROJCRS["WGS 84 / UTM zone 31N"

I also looked at any layers related to sentinel-5p, worldcover and Copernicus DEM:	line_number	collection_id	cell_size	comments
2199	TERRASCOPE_S5P_L3_NO2_TD_V1	(0.05;0.05)
2271	TERRASCOPE_S5P_L3_NO2_TM_V1	(0.05;0.05)
2343	TERRASCOPE_S5P_L3_NO2_TY_V1	(0.05;0.05)
2415	TERRASCOPE_S5P_L3_CO_TD_V1	(0.05;0.05)	GEOGCRS["WGS 84", DATUM["World Geodetic System 1984"
2487	TERRASCOPE_S5P_L3_CO_TM_V1	(0.05;0.05)
2556	TERRASCOPE_S5P_L3_CO_TY_V1	(0.05;0.05)
4250	SENTINEL_5P_L2		sentinel-hub
3505	ESA_WORLDCOVER_10M_2020_V1	(0.000083333333333;0.000083333333333)	/data/MTDA/WORLDCOVER/ESA_WORLDCOVER_10M_2020_V100/MAP
3569	ESA_WORLDCOVER_10M_2021_V2	(0.000083333333333;0.000083333333333)	/data/MTDA/WORLDCOVER/ESA_WORLDCOVER_10M_2021_V200/MAP
2982	COPERNICUS_30	(0.0002777777777777778;0.0002777777777777778)	/data/MTDA/DEM/COPERNICUS-DEM-30
3037	COPERNICUS_90	(0.000833333333;0.000833333333)	There is no /data/MTDA/DEM/COP_DEM_90M_COG on epod, only /data/MTDA/DEM/COPERNICUS-DEM-90. Is the root layer wrong in layercatalog.json?

I added an is_utm flag under the _vito key for SENTINEL1_GRD_SIGMA0 as an example: https://git.vito.be/projects/BIGGEO/repos/openeo-deploy/commits/dba691ada1a7e7e1fcedc9b3957f034f20442f4c#mep/layercatalog.json

@jdries Should I add this flag to all UTM layers?

[jdries]

yes, you can add the flag to utm layers, I'm only uncertain about sentinel-1 layers, as coherence had an issue so sigma0 maybe also.

[JeroenVerstraelen]

Do you mean that these layers could have errors when is_utm=True: SENTINEL1_GRD_SIGMA0 S1_GRD_SIGMA0_ASCENDING

So it would be best to write a test for them first and set is_utm=False if the test throws an error?

[JeroenVerstraelen]

Tested on dev using some simple load_collections:

import openeo

def main():
    for collection_id in ["TERRASCOPE_S2_FAPAR_V2", "TERRASCOPE_S2_NDVI_V2"]:
        bbox = {'west': 17.015762, 'east': 17.174721, 'south': 54.624568, 'north': 54.688916}
        temporal_extent = ["2022-09-02", "2022-09-02"]

        conn = openeo.connect("https://openeo-dev.vito.be/openeo/1.0/").authenticate_oidc()
        cube = conn.load_collection(collection_id = collection_id, spatial_extent = bbox, temporal_extent = temporal_extent)
        output_file = f"{collection_id}.tif"
        cube.download(output_file, format = "GTiff")

    collection_id = "TERRASCOPE_S1_SLC_COHERENCE_V1"
    temporal_extent = ["2021-03-20", "2021-03-20"]
    bbox = {'west': 4.398651, 'east': 4.590912, 'south': 50.975724, 'north': 51.060818}
    conn = openeo.connect("https://openeo-dev.vito.be/openeo/1.0/").authenticate_oidc()
    cube = conn.load_collection(collection_id = collection_id, spatial_extent = bbox, temporal_extent = temporal_extent)
    output_file = f"{collection_id}.tif"
    cube.download(output_file, format = "GTiff")

if __name__ == '__main__':
    main()