search

geoarrow / geoarrow-r

Extension types for geospatial data for use with 'Arrow'
http://geoarrow.org/geoarrow-r/
Apache License 2.0
152 stars 6 forks source link

GDAL RFC 86 Column-oriented read API for vector layers #23

Open rsbivand opened 2 years ago

rsbivand commented 2 years ago

https://lists.osgeo.org/pipermail/gdal-dev/2022-June/055884.html introduces https://github.com/OSGeo/gdal/pull/5830, https://github.com/rouault/gdal/blob/rfc_86/doc/source/development/rfc/rfc86_column_oriented_api.rst. Looks potentially promising! @edzer - is this helpful in sdsr?

jorisvandenbossche commented 2 years ago

Dewey is already working on a POC to consume it: https://github.com/paleolimbot/rfc86

paleolimbot commented 2 years ago

Joris beat me to it! I put that together yesterday because I was curious how much of the speed differences were internal to GDAL and how much of them were a result of (avoiding) constructing millions of R strings (my lightly tested conclusion was that it's some of both).

I think the main benefit will be that all the packages implementing a version of readOGR won't have to implement their own OGR field -> R vector logic. In that proof-of-concept I use the Arrow R package's conversion but there's no reason that the conversion couldn't live in something standalone (like narrow).