Element84 / earth-search

Earth Search information and issue tracking
https://earth-search.aws.element84.com/v1
27 stars 2 forks source link

Collection sentinel-2-l2c items have asset hrefs that reference sentinel-s2-l2a bucket #3

Open philvarner opened 1 year ago

philvarner commented 1 year ago

Collection sentinel-2-l2c items have asset hrefs that are incorrect. They reference non-existent files with the sentinel-s2-l2a bucket with the L1C path, e.g., S2A_39GWH_20230607_0_L1C references s3://sentinel-s2-l2a/tiles/39/G/WH/2023/6/7/0/B02.jp2

image
JonDHo commented 1 year ago

I just stumbled across this as well... it isn't all of the L1C items, but is a real problem. Is this going to be fixed??? Note also that the title of this issue has a typo - this relates to sentinel-2-l1c, not "l2c".

This is the example above in the actual catalog and it still has the L2A bucket referenced in the href items: https://earth-search.aws.element84.com/v1/collections/sentinel-2-l1c/items/S2A_39GWH_20230607_0_L1C

matthewhanson commented 1 year ago

@JonDHo Yes this will be fixed but I do not have a timeline. Currently we are updating the L2C files with the new processing baseline.

J6767 commented 1 year ago

Is there any workaround for this, please? Assume this is the same issue:

from pystac_client import Client

client = Client.open("https://earth-search.aws.element84.com/v1")
lat, lon = -26.954396644867618, 143.61560293259134

search = client.search(
    max_items=10,
    collections=['sentinel-2-l1c'],
    intersects=dict(type="Point", coordinates=[lon, lat]),
).get_all_items()

for i in range(len(items)):
    print(items[i].assets['red'].href)

s3://sentinel-s2-l1c/tiles/54/J/YR/2023/8/1/0/B04.jp2         # L1C
s3://sentinel-s2-l1c/tiles/54/J/YR/2023/7/27/0/B04.jp2
s3://sentinel-s2-l1c/tiles/54/J/YR/2023/7/22/0/B04.jp2
s3://sentinel-s2-l1c/tiles/54/J/YR/2023/7/17/0/B04.jp2
s3://sentinel-s2-l1c/tiles/54/J/YR/2023/7/12/0/B04.jp2
s3://sentinel-s2-l2a/tiles/54/J/YR/2023/7/7/0/B04.jp2        # L2A
s3://sentinel-s2-l2a/tiles/54/J/YR/2023/7/2/0/B04.jp2
s3://sentinel-s2-l2a/tiles/54/J/YR/2023/6/27/0/B04.jp2
s3://sentinel-s2-l2a/tiles/54/J/YR/2023/6/22/0/B04.jp2
s3://sentinel-s2-l2a/tiles/54/J/YR/2023/6/17/0/B04.jp2

Our processing modality requires L1C products only. When used with Stackstac, Rasterio is erroring: RuntimeError: Error opening 's3://sentinel-s2-l2a/tiles/54/J/YR/2023/7/7/0/B02.jp2': RasterioIOError('The specified key does not exist.')

Many thanks.

matthewhanson commented 1 year ago

@J6767 The workaround is to loop through the assets and replace the string:

for i in items:
   for a in i['assets']:
      i['assets'][a]['href'] = i['assets'][a]['href'].replace('sentinel-s2-l2a', 'sentinel-s2-l1c')

This has been fixed on new scenes going forward.

J6767 commented 1 year ago

Thanks very much, I just wasn't sure if the products were in the correct bucket.