Failed to retrieve some Landsat8 data

[BasileGoussard]

Firstly thanks for your amazing work! Netcarbon is part of PANGEO Europe and we are so pleased to work with Planetary Computer.

We regularly face the same issue with the Landsat dataset on Microsoft Planetary Computer. It seems to miss some data on the Landsat bucket compared to what the output of the STAC API.

Here is an example:

• We search landsat with stac_client
• We create datacube by using stackstac
• We apply algorithms by using dask and xarray
• We regularly face the following issue (one raster data seem to not be store in the relevant way in the bucket)

RuntimeError: Error opening 'https://landsateuwest.blob.core.windows.net/landsat-c2/level-2/standard/oli-tirs/2022/199/030/LC09_L2SP_199030_20220704_20220803_02_T1/LC09_L2SP_199030_20220704_20220803_02_T1_SR_B5.TIF?st=2023-06-26T09%3A35%3A55Z&se=2023-06-27T10%3A20%3A55Z&sp=rl&sv=2021-06-08&sr=c&skoid=c85c15d6-d1ae-42d4-af60-e2ca0f81359b&sktid=72f988bf-86f1-41af-91ab-2d7cd011db47&skt=2023-06-27T08%3A29%3A49Z&ske=2023-07-04T08%3A29%3A49Z&sks=b&skv=2021-06-08&sig=K%2Bmm3czlSviBTkyXvbYE0MPYrA/61x5vHA2YGvqft9Y%3D': RasterioIOError("'/vsicurl/https://landsateuwest.blob.core.windows.net/landsat-c2/level-2/standard/oli-tirs/2022/199/030/LC09_L2SP_199030_20220704_20220803_02_T1/LC09_L2SP_199030_20220704_20220803_02_T1_SR_B5.TIF?st=2023-06-26T09%3A35%3A55Z&se=2023-06-27T10%3A20%3A55Z&sp=rl&sv=2021-06-08&sr=c&skoid=c85c15d6-d1ae-42d4-af60-e2ca0f81359b&sktid=72f988bf-86f1-41af-91ab-2d7cd011db47&skt=2023-06-27T08%3A29%3A49Z&ske=2023-07-04T08%3A29%3A49Z&sks=b&skv=2021-06-08&sig=K%2Bmm3czlSviBTkyXvbYE0MPYrA/61x5vHA2YGvqft9Y%3D' not recognized as a supported file format.")

Looking for your feedback, Best Basile

[TomAugspurger]

See https://github.com/microsoft/PlanetaryComputer/discussions/101 for some more information, but the tl/dr is that for some older scenes we didn't properly handle some of the responses from USGS's HTTP servers. This is fixed for all newer scenes. At some point we'll get around to fixing all those, but I don't have a timeline for that.

[BasileGoussard]

Hi Tom, thanks for your answer. Do you know how to still build a datacube without the missing scenes when using stackstac ? @gjoseph92

[TomAugspurger]

That's a tricky one... there's nothing in the STAC metadata indicating that some of the assets might be missing, and stackstac is based purely off the STAC metadata.

I suspect the errors_as_nodata keyword to stackstac.stack should work for you: https://stackstac.readthedocs.io/en/latest/api/main/stackstac.stack.html. I'm not sure exactly what to set it to, but it will turn those exceptions into missing values when you're actually reading the data.

[TomAugspurger]

Closing this in favor of #101.

Some of these will have been fixed, but will have different paths in blob storage. See https://github.com/microsoft/PlanetaryComputer/discussions/101#discussioncomment-6939924 for details.