Open vincentsarago opened 2 years ago
Yes - had this happen this week - using a dask Fargate cluster and occasionally otherwise something that reads fine then stops - and possibly when reading chunks of same file it craps out.
e.g. use a 1 machine cluster, ok for sometime - use 100 reading lots of parts, instant failure.
👋 rio-tiler-fs is an experimental plugin to create a custom COGReader that uses fsspec to open dataset instead of relying on GDAL.
This is experimental because file-like VSI plugin is not yet publish in a stable rasterio version
How To
Within the custom COGReader we are opening the input file (url) using fsspec and then passing it to rasterio.
Why
We have seen numerous reports of issue when using S3 files (mostly when using threads) where for unknown reason (to me) GDAL/Rasterio will loose the ability to read a file.
A second reason will be to enable custom cache 🤷 ref: https://github.com/gjoseph92/stackstac/issues/90