search

cogeotiff / rio-tiler-fs

[EXPERIMENTAL/WIP] Connect rio-tiler and fsspec
MIT License
3 stars 1 forks source link

Issue 1 #1

Open vincentsarago opened 2 years ago

vincentsarago commented 2 years ago

👋 rio-tiler-fs is an experimental plugin to create a custom COGReader that uses fsspec to open dataset instead of relying on GDAL.

This is experimental because file-like VSI plugin is not yet publish in a stable rasterio version

How To

Within the custom COGReader we are opening the input file (url) using fsspec and then passing it to rasterio. 

# rio_tiler.io.cogeo.COGReader
with rasterio.open(src_path) as src:
    data = src.read()

# rio_tiler_fs.reader.COGReader
with fsspec.open(src_path) as fsobj:
    with rasterio.open(fsobj) as src:
        data = src.read()

Why

We have seen numerous reports of issue when using S3 files (mostly when using threads) where for unknown reason (to me) GDAL/Rasterio will loose the ability to read a file.

A second reason will be to enable custom cache 🤷 ref: https://github.com/gjoseph92/stackstac/issues/90

RichardScottOZ commented 2 years ago

Yes - had this happen this week - using a dask Fargate cluster and occasionally otherwise something that reads fine then stops - and possibly when reading chunks of same file it craps out.

e.g. use a 1 machine cluster, ok for sometime - use 100 reading lots of parts, instant failure.