geopandas / pyogrio

Vectorized vector I/O using OGR
https://pyogrio.readthedocs.io
MIT License
257 stars 21 forks source link

ENH: enable support for writing to memory #397

Closed brendan-ward closed 2 months ago

brendan-ward commented 2 months ago

Resolves https://github.com/geopandas/pyogrio/issues/249

This enables basic writing to memory using an io.BytesIO object. This does not currently support append / adding layers; I tried working that out unsuccessfully in earlier commits, but we could return to this in a later PR. It seems like it should be possible in theory.

This specifically blacklists ESRI Shapefile and OpenFileGDB because they write multiple files, which can't be returned successfully as a single memory file.

jorisvandenbossche commented 2 months ago

Some things we still need to add / figure out:

  • append / add layers (still trying to figure out how to make this work)

FWIW, I would personally first focus on getting the plain writing to memory work without append (just raise for now if the mode is append)
(fiona also raises an error if mode="a" for writing to file-like objects)

brendan-ward commented 2 months ago

Fiona has specific tests for appending so I assumed it was possible and expected behavior, but indeed in the interest of getting this out sooner we can more strictly limit this to write-only mode.

jorisvandenbossche commented 2 months ago

Fiona has specific tests for appending so I assumed it was possible and expected behavior

I assume this only works when directly using the MemoryFile interface. Because at that point you have the same object, where you first write to, and then append to. But when creating a new memfile, you can't append to it (test). And that is essentially what happens in geopandas when people pass a BytesIO object to to_file, we each time create a new MemoryFile object through fiona.open(), and thus only pure writing is supported (fiona explicitly raise in case of append mode in open() for a writeable object).

So longer term that is also something we could do, but it might requiring creating some similar custom object like fiona's MemoryFile?