search

intake / intake_geopandas

An intake plugin for loading datasets with geopandas
BSD 2-Clause "Simplified" License
15 stars 7 forks source link

Enable caching for file-like sources. #10

Closed ian-r-rose closed 4 years ago

ian-r-rose commented 4 years ago

This enables caching for shapefiles and GeoJSON.

martindurant commented 4 years ago

This is fine with me, but following the appearance of cacheing in fsspec, you may find the new alternative easier to use. I mean to write a blog post about why this is good with or without Intake.

ian-r-rose commented 4 years ago

Ah, interesting. To be honest, I haven't really thought through the handoff between using fsspec aind GDAL/OGR/fiona for this. The url path logic is meaningfully different, so there are likely places where I need to be a bit more careful...

martindurant commented 4 years ago

If you are relying on GDAL, then you want the whole-file cacheing variant, which is what you will have wanted Intake to do. In that case, you get a local copy of the file, and pass off the real open file (or path). Note that with the new fsspec implementation, arguments go into the normal set of arguments, so you can use normal Intake magics like CATALOG_DIR or environment variables to specify the cache location (or just have a temporary dir which lives as long as the current process).

ian-r-rose commented 4 years ago

Closing this for now until I have time to more carefully think through what the behavior should be.