search

opendatacube / odc-stac

Load STAC items into xarray Datasets.
Apache License 2.0
140 stars 20 forks source link

Allow errors besides RasterioIOError in odc.stac.load with fail_on_error=False #122

Closed jessjaco closed 11 months ago

jessjaco commented 1 year ago

I'm getting an error while reading a tiff that starts with rasterio._err.CPLE_AppDefinedError: IReadBlock failed at X offset 9, Y offset 3: TIFFReadEncodedTile() failed etc. and ends with rasterio.errors.WarpOperationError: Chunk and warp failed. I can send a more detailed example but ultimately this is a bad data file, and I just want to ignore the error and go on to the next file.

I see here https://github.com/opendatacube/odc-stac/blob/52a016be2115f180e7059f67bcc6106fbeba7e8d/odc/stac/_reader.py#L187 that this specific error is not ignored. Would it be possible to set the specific errors that could be ignored, or something similar? As an example, stackstac.stack has an errors_as_nodata option, which allows specific errors to be defined.

Kirill888 commented 1 year ago

Hm, I think that code should just catch a more generic rasterio.errors.RasterioError, or even Exception.

I'm not sure about the value of capturing user configured exception types, this might be reasonable if we had support for multiple IO backends, but we only have rasterio backend right-now.

jessjaco commented 1 year ago

Ok, makes sense I'm testing it out,

Kirill888 commented 1 year ago

thanks @jessjaco, happy to accept a PR if that works for you.

And for a more generic handling of errors I'd rather go with a user supplied callback, although Dask adds some constraints on that approach.