More space efficient object encoding

mapbox / osm-wayback

Scalable RocksDB index from OSM planet to lookup historic OSM objects.

BSD 3-Clause "New" or "Revised" License

2 stars 2 forks source link

More space efficient object encoding #25

Open lukasmartinelli opened 7 years ago

lukasmartinelli commented 7 years ago

Due to fast prototyping we were just serializing the RapidJSON object into a string. I think even though RocksDB gets rid of a lot of the overhead with compression if we want to get serious about file size it makes sense to encode the data better.

Using protozero and a definition like https://github.com/scrosby/OSM-binary/blob/master/src/osmformat.proto we can store the objects and tags much more space efficient.

jenningsanderson commented 7 years ago

Putting a plug here for geobuf (which I believe is the next iteration of osm-binary?); but now that tippecanoe natively supports... we're going to be flying!