search

peermaps / ingest

Convert osm pbf files into a format that can be used in peermaps.
15 stars 1 forks source link

swap out o5m for json #8

Closed okdistribute closed 3 years ago

okdistribute commented 3 years ago

swap out o5m for json

maybe we could gzip the json at least. if we can use the browser's built-in gzip decompressor that would be ideal but otherwise we can probably use a wasm un-gzip

ghost commented 3 years ago

I think the o5m is also causing a huge inflation in the output file size when I run the ingest on a small metro extract (I stopped it after it wrote a gigabyte for a 7M file). My guess is that the persistent o5m string table is getting written every time but that only makes sense when there are many o5m records.

ghost commented 3 years ago

This has been addressed in the leveldb code which uses a compact binary encoder and decoder to store intermediate representations of osm features.