Open stevage opened 5 years ago
Unfortunately it is very difficult to avoid having all the data in memory with protocol buffers, because the header for each object includes its length. (It would still be better to have the whole protocol buffer in memory than to have the whole GeoJSON in memory, though.)
@ericfischer I agree but I think the Readme is not clear on this
The encoding format also potentially allows:
Easy incremental parsing — get features out as you read them, without the need to build in-memory representation of the whole data.
Partial reads — read only the parts you actually need, skipping the rest.
As far as I can tell the current proto file data message is by design that you have to put everything in one proto message right? So that contradicts easy incremental parsing, correct?
It would be really handy if the command line tools could support newline-delimited GeoJSON, similar to Tippecanoe, shp2json and the ndjson-cli library.
In theory, processing newline-delimited files means memory is not a constraint, as the entire source GeoJSON doesn't need to be read into memory before processing begins. But I don't know enough about PBF to know whether that's actually true in this instance.