Read or write failed. IReadBlock failed at X offset 0, Y offset 0: IReadBlock failed at X offset 20, Y offset 12: TIFFReadEncodedTile() failed

[mehran66]

I sometimes get the following error when running the stackstac.stack. This error does not occur all of the times while running the same code! For now, I just implemented a retry mechanism.

data = (
    stackstac.stack(
        items,
        assets=["B02", "B03", "B04", "B08"], 
        chunksize=2304,
        resolution=10,
        epsg=most_common_epsg,
        bounds_latlon=bbox,
    )
    .where(lambda x: x > 0, other=np.nan)  # sentinel-2 uses 0 as nodata
    .assign_coords(band=lambda x: x.common_name.rename("band"))  # use common names
)

Error reading Window(col_off=0, row_off=0, width=1950, height=1950) from 'https://sentinel2l2a01.blob.core.windows.net/sentinel2-l2/11/S/LT/2017/06/13/S2A_MSIL2A_20170613T182921_N0212_R027_T11SLT_20210209T224230.SAFE/GRANULE/L2A_T11SLT_A010320_20170613T183355/IMG_DATA/R10m/T11SLT_20170613T182921_B04_10m.tif?st=2023-10-02T19%3A26%3A13Z&se=2023-10-03T20%3A11%3A13Z&sp=rl&sv=2021-06-08&sr=c&skoid=c85c15d6-d1ae-42d4-af60-e2ca0f81359b&sktid=72f988bf-86f1-41af-91ab-2d7cd011db47&skt=2023-10-03T17%3A11%3A45Z&ske=2023-10-10T17%3A11%3A45Z&sks=b&skv=2021-06-08&sig=uo6P2FvIfahk3On4ADj8egCudxKiGsSaX4j/hmdn2WQ%3D': RasterioIOError('Read or write failed. IReadBlock failed at X offset 0, Y offset 0: IReadBlock failed at X offset 20, Y offset 12: TIFFReadEncodedTile() failed.')

Also sometimes, I face other similar errors that get resolved after retrying:

Error opening 'https://sentinel2l2a01.blob.core.windows.net/sentinel2-l2/10/S/EH/2020/08/22/S2A_MSIL2A_20200822T184921_N0212_R113_T10SEH_20200825T124924.SAFE/GRANULE/L2A_T10SEH_A026994_20200822T190121/IMG_DATA/R10m/T10SEH_20200822T184921_B08_10m.tif?st=2023-10-02T20%3A57%3A02Z&se=2023-10-03T21%3A42%3A02Z&sp=rl&sv=2021-06-08&sr=c&skoid=c85c15d6-d1ae-42d4-af60-e2ca0f81359b&sktid=72f988bf-86f1-41af-91ab-2d7cd011db47&skt=2023-10-03T13%3A31%3A17Z&ske=2023-10-10T13%3A31%3A17Z&sks=b&skv=2021-06-08&sig=7J/j2FJjYw3pdyGyZXFeDbzrJip7fwmmCiDWF2lbbtM%3D': RasterioIOError('HTTP response code: 503')

Error opening 'https://sentinel2l2a01.blob.core.windows.net/sentinel2-l2/11/S/MS/2020/12/19/S2B_MSIL2A_20201219T183759_N0212_R027_T11SMS_20201220T165730.SAFE/GRANULE/L2A_T11SMS_A019787_20201219T184458/IMG_DATA/R10m/T11SMS_20201219T183759_B08_10m.tif?st=2023-10-03T13%3A35%3A38Z&se=2023-10-04T14%3A20%3A38Z&sp=rl&sv=2021-06-08&sr=c&skoid=c85c15d6-d1ae-42d4-af60-e2ca0f81359b&sktid=72f988bf-86f1-41af-91ab-2d7cd011db47&skt=2023-10-04T13%3A24%3A10Z&ske=2023-10-11T13%3A24%3A10Z&sks=b&skv=2021-06-08&sig=FyHq7ke3q%2BmAaVvIScoez1S/foziLy8SX0QKohxlGtY%3D': RasterioIOError("'/vsicurl/https://sentinel2l2a01.blob.core.windows.net/sentinel2-l2/11/S/MS/2020/12/19/S2B_MSIL2A_20201219T183759_N0212_R027_T11SMS_20201220T165730.SAFE/GRANULE/L2A_T11SMS_A019787_20201219T184458/IMG_DATA/R10m/T11SMS_20201219T183759_B08_10m.tif?st=2023-10-03T13%3A35%3A38Z&se=2023-10-04T14%3A20%3A38Z&sp=rl&sv=2021-06-08&sr=c&skoid=c85c15d6-d1ae-42d4-af60-e2ca0f81359b&sktid=72f988bf-86f1-41af-91ab-2d7cd011db47&skt=2023-10-04T13%3A24%3A10Z&ske=2023-10-11T13%3A24%3A10Z&sks=b&skv=2021-06-08&sig=FyHq7ke3q%2BmAaVvIScoez1S/foziLy8SX0QKohxlGtY%3D' not recognized as a supported file format.")

[TomAugspurger]

See https://github.com/microsoft/PlanetaryComputer/issues/11#issuecomment-995380986. You'll want to set GDAL_HTTP_MAX_RETRY and GDAL_HTTP_RETRY_DELAY to ensure that your code retries those failures.

xref https://github.com/gjoseph92/stackstac/issues/18 (maybe you want to submit a PR to fix that?)

[mehran66]

Thanks @TomAugspurger

I added both ways of setting GDAL_HTTP_MAX_RETRY and GDAL_HTTP_RETRY_DELAY and the issue got resolved. Not sure which one helps but does not bother to have both :)

os.environ["GDAL_HTTP_MAX_RETRY"] = "5"
os.environ["GDAL_HTTP_RETRY_DELAY"] = "1"

data = (
    stackstac.stack(
        items,
        assets=["B02", "B03", "B04", "B08"], 
        chunksize=2304,
        resolution=10,
        epsg=most_common_epsg,
        bounds_latlon=bbox,
        gdal_env=stackstac.DEFAULT_GDAL_ENV.updated(always=dict(GDAL_HTTP_MAX_RETRY=5, GDAL_HTTP_RETRY_DELAY=1))
    )
    .where(lambda x: x > 0, other=np.nan)
    .assign_coords(band=lambda x: x.common_name.rename("band"))
)

[thaimunhoz]

Hi, I'm having the same problem, but I'm currently using stac.load. Do you guys know how I can set this parameter on stac_load?

gdal_env=stackstac.DEFAULT_GDAL_ENV.updated(always=dict(GDAL_HTTP_MAX_RETRY=5, GDAL_HTTP_RETRY_DELAY=1))

[TomAugspurger]

Looks like odc.stac.configure_rio might be what you want: https://odc-stac.readthedocs.io/en/latest/_api/odc.stac.configure_rio.html?highlight=GDAL#odc.stac.configure_rio (untested).

Also worth confirming that you are indeed getting 500 errors, and not some other status code (403, or 404) which would indicate either a problem with the data in Blob Storage or an issue with your SAS token.