search

mapbox / geobuf

A compact binary encoding for geographic data.
ISC License
967 stars 84 forks source link

Support newline-delimited GeoJSON #103

Open stevage opened 5 years ago

stevage commented 5 years ago

It would be really handy if the command line tools could support newline-delimited GeoJSON, similar to Tippecanoe, shp2json and the ndjson-cli library.

In theory, processing newline-delimited files means memory is not a constraint, as the entire source GeoJSON doesn't need to be read into memory before processing begins. But I don't know enough about PBF to know whether that's actually true in this instance.

e-n-f commented 5 years ago

Unfortunately it is very difficult to avoid having all the data in memory with protocol buffers, because the header for each object includes its length. (It would still be better to have the whole protocol buffer in memory than to have the whole GeoJSON in memory, though.)

gertcuykens commented 5 years ago

@ericfischer I agree but I think the Readme is not clear on this 

The encoding format also potentially allows:
Easy incremental parsing — get features out as you read them, without the need to build in-memory representation of the whole data.
Partial reads — read only the parts you actually need, skipping the rest.

As far as I can tell the current proto file data message is by design that you have to put everything in one proto message right? So that contradicts easy incremental parsing, correct?