search

Open-EO / openeo-geopyspark-driver

OpenEO driver for GeoPySpark (Geotrellis)
Apache License 2.0
25 stars 4 forks source link

sentinel3: error on LST + solar angles #755

Open jdries opened 2 months ago

jdries commented 2 months ago

Getting error:

E                   reprojected_data = apply_LUT_on_band(band_data, LUT_angles, band_settings.get('_FillValue', None))  # result is an numpy array with reprojected data
E                 File "/opt/openeo/lib/python3.8/site-packages/openeogeotrellis/collections/sentinel3.py", line 638, in apply_LUT_on_band
E                   grid_values = data_flat[LUT]
E               IndexError: index 50018 is out of bounds for axis 0 with size 46743

On this load_collection:

datacube = con.load_collection(
        "SENTINEL3_SLSTR_L2_LST",
        spatial_extent={"west": -95, "south": 40, "east": -93, "north": 44},
        temporal_extent=["2021-08-03", "2021-08-04"],
        bands=["sunZenithAngles"]
    )
jdries commented 2 months ago

Extra message reveals more info:

openeo_driver.errors.InternalException: Server error: Failed to read Sentinel-3 SL_2_LST___ ['geometry_tn:solar_zenith_tn'] for /eodata/Sentinel-3/SLSTR/SL_2_LST/2021/08/03/S3B_SL_2_LST____20210803T171740_20210803T172040_20210805T033653_0179_055_226_2160_LN2_O_NT_004.SEN3 and extent {'xmin': -95.0, 'xmax': -92.71428571428582, 'ymin': 39.42857142857165, 'ymax': 44.0}. Error: index 10858 is out of bounds for axis 0 with size 8709

STAC item: https://catalogue.dataspace.copernicus.eu/stac/collections/SENTINEL-3/items/S3B_SL_2_LST____20210803T171740_20210803T172040_20210805T033653_0179_055_226_2160_LN2_O_NT_004.SEN3

jdries commented 2 months ago

Product geometry vs bbox reveals that this could be an edge case:

Image

{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "properties": {},
      "geometry": {
        "coordinates": [
          [
            [
              -95.0,
              44.0
            ],
            [
              -95.0,
              40.0
            ],
            [
              -93.0,
              40.0
            ],
            [
              -93.0,
              44.0
            ],
            [
              -95.0,
              44.0
            ]
          ]
        ],
        "type": "Polygon"
      }
    },
    {
"type": "Feature",
"stac_version": "1.0.0",
"stac_extensions": [
"https://stac-extensions.github.io/alternate-assets/v1.1.0/schema.json",
"https://stac-extensions.github.io/storage/v1.0.0/schema.json"
],
"id": "S3B_SL_2_LST____20210803T171740_20210803T172040_20210805T033653_0179_055_226_2160_LN2_O_NT_004.SEN3",
"collection": "SENTINEL-3",
"geometry": {
"type": "Polygon",
"coordinates": [
[
[
-114.446,
41.9979
],
[
-113.835,
41.9437
],
[
-113.223,
41.8813
],
[
-112.615,
41.832
],
[
-112.006,
41.7569
],
[
-111.399,
41.6878
],
[
-110.797,
41.6232
],
[
-110.194,
41.557
],
[
-109.59,
41.4725
],
[
-108.996,
41.3972
],
[
-108.388,
41.3144
],
[
-107.794,
41.2236
],
[
-107.192,
41.1338
],
[
-106.602,
41.0446
],
[
-106.005,
40.9466
],
[
-105.423,
40.8503
],
[
-104.833,
40.7466
],
[
-104.239,
40.6481
],
[
-103.656,
40.5401
],
[
-103.067,
40.4274
],
[
-102.493,
40.3144
],
[
-101.909,
40.1961
],
[
-101.334,
40.0759
],
[
-100.753,
39.9523
],
[
-100.176,
39.8342
],
[
-99.6152,
39.7039
],
[
-99.0427,
39.5698
],
[
-98.4786,
39.4396
],
[
-97.9148,
39.3049
],
[
-97.3413,
39.1668
],
[
-97.1347,
39.1133
],
[
-96.0241,
41.6625
],
[
-94.8156,
44.2461
],
[
-93.5001,
46.8159
],
[
-92.0627,
49.3612
],
[
-92.0705,
49.3729
],
[
-92.3063,
49.4308
],
[
-92.9793,
49.5937
],
[
-93.6318,
49.7519
],
[
-94.3052,
49.9124
],
[
-94.9716,
50.0588
],
[
-95.6474,
50.2091
],
[
-96.3302,
50.3548
],
[
-97.0072,
50.4951
],
[
-97.6971,
50.6229
],
[
-98.3976,
50.7567
],
[
-99.0858,
50.8836
],
[
-99.7808,
51.0065
],
[
-100.484,
51.1262
],
[
-101.197,
51.2436
],
[
-101.898,
51.3517
],
[
-102.619,
51.461
],
[
-103.33,
51.5603
],
[
-104.061,
51.6612
],
[
-104.776,
51.7479
],
[
-105.506,
51.8349
],
[
-106.224,
51.9165
],
[
-106.962,
52.0002
],
[
-107.707,
52.0784
],
[
-108.437,
52.1472
],
[
-109.163,
52.2103
],
[
-109.9,
52.2672
],
[
-110.645,
52.335
],
[
-111.394,
52.3781
],
[
-112.122,
52.4285
],
[
-112.887,
52.4686
],
[
-112.888,
52.4599
],
[
-113.266,
49.8424
],
[
-113.654,
47.2146
],
[
-114.049,
44.5852
],
[
-114.446,
41.9979
]
]
]
},
"properties": {
"authority": "ESA",
"timeliness": "NT",
"coordinates": "41.9979,-114.446 41.9437,-113.835 41.8813,-113.223 41.832,-112.615 41.7569,-112.006 41.6878,-111.399 41.6232,-110.797 41.557,-110.194 41.4725,-109.59 41.3972,-108.996 41.3144,-108.388 41.2236,-107.794 41.1338,-107.192 41.0446,-106.602 40.9466,-106.005 40.8503,-105.423 40.7466,-104.833 40.6481,-104.239 40.5401,-103.656 40.4274,-103.067 40.3144,-102.493 40.1961,-101.909 40.0759,-101.334 39.9523,-100.753 39.8342,-100.176 39.7039,-99.6152 39.5698,-99.0427 39.4396,-98.4786 39.3049,-97.9148 39.1668,-97.3413 39.1133,-97.1347 41.6625,-96.0241 44.2461,-94.8156 46.8159,-93.5001 49.3612,-92.0627 49.3729,-92.0705 49.4308,-92.3063 49.5937,-92.9793 49.7519,-93.6318 49.9124,-94.3052 50.0588,-94.9716 50.2091,-95.6474 50.3548,-96.3302 50.4951,-97.0072 50.6229,-97.6971 50.7567,-98.3976 50.8836,-99.0858 51.0065,-99.7808 51.1262,-100.484 51.2436,-101.197 51.3517,-101.898 51.461,-102.619 51.5603,-103.33 51.6612,-104.061 51.7479,-104.776 51.8349,-105.506 51.9165,-106.224 52.0002,-106.962 52.0784,-107.707 52.1472,-108.437 52.2103,-109.163 52.2672,-109.9 52.335,-110.645 52.3781,-111.394 52.4285,-112.122 52.4686,-112.887 52.4599,-112.888 49.8424,-113.266 47.2146,-113.654 44.5852,-114.049 41.9979,-114.446",
"orbitNumber": 17053,
"productType": "SL_2_LST___",
"orbitDirection": "DESCENDING",
"operationalMode": "Earth Observation",
"processingLevel": "2",
"platformShortName": "SENTINEL-3",
"baselineCollection": "004",
"instrumentShortName": "SLSTR",
"relativeOrbitNumber": 226,
"platformSerialIdentifier": "B",
"datetime": "2021-08-03T17:17:40.320Z",
"end_datetime": "2021-08-03T17:20:40.320Z",
"start_datetime": "2021-08-03T17:17:40.320Z"
},
"bbox": [
-114.44600677490234,
39.11329650878906,
-92.06269836425781,
52.46860122680664
],
"links": [
{
"rel": "root",
"type": "application/json",
"href": "https://catalogue.dataspace.copernicus.eu/stac"
},
{
"rel": "self",
"type": "application/json",
"href": "https://catalogue.dataspace.copernicus.eu/stac/collections/SENTINEL-3/items/S3B_SL_2_LST____20210803T171740_20210803T172040_20210805T033653_0179_055_226_2160_LN2_O_NT_004.SEN3"
},
{
"rel": "collection",
"type": "application/json",
"href": "https://catalogue.dataspace.copernicus.eu/stac/collections/SENTINEL-3"
}
],
"assets": {
"QUICKLOOK": {
"href": "https://catalogue.dataspace.copernicus.eu/odata/v1/Assets(79afeba1-284c-4706-a67e-c7b0f4d230bd)/$value",
"title": "QUICKLOOK",
"type": "image/jpeg"
},
"PRODUCT": {
"href": "https://catalogue.dataspace.copernicus.eu/odata/v1/Products(bbb92f3e-25c3-5a1a-9129-8d5281f726d3)/$value",
"title": "Product",
"type": "application/octet-stream",
"alternate": {
"s3": {
"href": "/eodata/Sentinel-3/SLSTR/SL_2_LST/2021/08/03/S3B_SL_2_LST____20210803T171740_20210803T172040_20210805T033653_0179_055_226_2160_LN2_O_NT_004.SEN3",
"storage:platform": "CLOUDFERRO",
"storage:region": "waw",
"storage:requester_pays": false,
"storage:tier": "Online"
}
}
}
}
}
  ]
}
jdries commented 2 months ago

Note: the issue is clearly specific to angle bands, and how they build a separate LUT. I pushed a fix that avoids throwing the error, but I believe data will be incorrect, so not yet much better.

bossie commented 2 weeks ago

The result for this process graph looks a little funky in QGIS but it's actually two gradients stitched together. If you adjust the value range a bit you can see this: original_top_gradient original_bottom_gradient

The gradients don't match up + features from multiple orbits suggest that it's a matter of narrowing the time range.

bossie commented 2 weeks ago

Narrowing the time range to ["2021-08-03T04:02:00Z", "2021-08-03T04:10:00Z"] yields only two adjacent features from within the same orbit (reddish); in the background are the two corresponding sunZenithAngles source assets (georeferenced manually) and in the foreground is what OpenEO now creates for this AOI (its values match the those of the source assets but I adjusted the range in QGIS to make it stand out):

narrower_time_range_gradient

bossie commented 2 weeks ago

Got an integration test ready as well.

bossie commented 2 weeks ago

Said integration test test_collections.test_compare_SENTINEL3_SLSTR_L2_LST_solar_angles_large passes.

bossie commented 2 weeks ago

Still needs work: 3 out of 4 output assets for the original process graph look good but the last one is wrong:

Image Image Image Image