georust / geozero

Zero-Copy reading and writing of geospatial data.
Apache License 2.0
356 stars 38 forks source link

Implement a streaming GeoJSON reader #1

Closed pka closed 8 months ago

pka commented 4 years ago

The current GeoJSON reader is based on https://github.com/georust/geojson which reads the whole file into memory.

A few unsuccessful attempts are in https://github.com/pka/geozero/tree/parse-geojson-with-serde:

I also had a deeper look at https://github.com/Geal/nom, which has a streaming interface. But I didn't find a good way to continue after a "Buffer empty" error.

frewsxcv commented 4 years ago

a couple relevant issues in the popular json libraries:

pka commented 3 years ago

Apache Arrow has a batch reader: https://github.com/apache/arrow/blob/master/rust/arrow/src/json/reader.rs

pka commented 3 years ago

Rust library for encoding and decoding JSON streams.: https://docs.rs/destream_json/

pka commented 3 years ago

https://github.com/jeremiah-shaulov/nop-json: Deserialization of JSON io::Read stream into primitive types, Strings and any other types.

EDIT: Currently nighly only.

pka commented 3 years ago

https://github.com/serde-rs/json-benchmark/blob/master/src/canada.rs contains a GeoJSON benchmark.

bjornharrtell commented 2 years ago

Arrow implementation is now at https://github.com/apache/arrow-rs/blob/master/arrow/src/json/reader.rs.

kylebarron commented 8 months ago

Arrow implementation is now at https://github.com/apache/arrow-rs/blob/master/arrow-json/src/reader/mod.rs 🙃

kylebarron commented 8 months ago

Was this issue intended to be closed by #16?

michaelkirk commented 8 months ago

I think so!