BBBike uses osmconvert to generate PBFs. Reading through the osmconvert code 1, osmconvert will pack up to 31 MB of data into each block.
PBFs from Geofabrik use Osmium, which defaults to packing 8,000 objects into each block, resulting in blocks that are more like ~60 KB in size.
More, smaller blocks are better for Tilemaker:
fewer blocks means less opportunity to use multiple cores
larger blocks means higher baseline memory requirement
I emailed BBBike's maintainer. He wants to one day move to Osmium, and is understandably not keen on patching osmconvert in the interim.
He did point out that a user can just run osmium cat on a PBF from BBBike to rejig its innards.
This PR detects when a PBF is suboptimal, warns the user, and provides an explanation of how to fix it.
For one of my BBBike PBFs, this results in processing time dropping from 90 seconds to 28 seconds. (osmium cat itself only takes 17 seconds, and in any case, only has to be run a single time.)
Tilemaker's documentation suggests that users can get PBFs from BBBike: https://github.com/systemed/tilemaker/blob/1da4be97dc4ac7f11daf3f417c7ca0a6a34ae47f/docs/VECTOR_TILES.md?plain=1#L25
BBBike uses osmconvert to generate PBFs. Reading through the osmconvert code 1, osmconvert will pack up to 31 MB of data into each block.
PBFs from Geofabrik use Osmium, which defaults to packing 8,000 objects into each block, resulting in blocks that are more like ~60 KB in size.
More, smaller blocks are better for Tilemaker:
I emailed BBBike's maintainer. He wants to one day move to Osmium, and is understandably not keen on patching osmconvert in the interim.
He did point out that a user can just run
osmium cat
on a PBF from BBBike to rejig its innards.This PR detects when a PBF is suboptimal, warns the user, and provides an explanation of how to fix it.
For one of my BBBike PBFs, this results in processing time dropping from 90 seconds to 28 seconds. (
osmium cat
itself only takes 17 seconds, and in any case, only has to be run a single time.)