search

mapbox / geobuf

A compact binary encoding for geographic data.
ISC License
968 stars 84 forks source link

Stream multiple geobuf messages #47

Closed fnogatz closed 9 years ago

fnogatz commented 9 years ago

I have a toolchain where I pipe GeoJSON documents as line delimited JSON from one to another. It would be nice to encode the single GeoJSON instances as geobuf and pipe those around too.

My first try was to use binary-split to split a stream into several geobuf instances. Unfortunately you have to use a special splitOn ASCII sequence I don't know as the default linebreak could be part of a geobuf too.

So is there any ASCII sequence which is significant for a geobuf start or ending? I know of #37, but there only the streaming mode of a single GeoJSON document has been discussed.

mourner commented 9 years ago

Sorry for a late response. I think it's not possible to use binary-split technique with Procol Buffers for streaming — see https://developers.google.com/protocol-buffers/docs/techniques for reasons why and possible workarounds.

mourner commented 9 years ago

Also see this https://github.com/mafintosh/protocol-buffers-stream

mourner commented 9 years ago

Got my own low-level protobuf splitter working: https://github.com/mapbox/pbf-split (readme/tests to follow)