systemed / tilemaker

Make OpenStreetMap vector tiles without the stack
https://tilemaker.org/
Other
1.44k stars 229 forks source link

mbtiles: only create 1 unique index on tiles #603

Closed cldellow closed 8 months ago

cldellow commented 9 months ago

Before:

After:

This reduces the mbtiles file by 0.5% and processing time by perhaps 0.3%, although it's quite noisy.

Creating the index at the start, rather than at the end, seemed to be fastest.

systemed commented 8 months ago

Good spot! Usual wisdom is to create the index at the end, but because we run all the inserts in one transaction, it should be fine to do it at the start.

cldellow commented 8 months ago

Yeah, I think it depends what you want to optimize for. I would expect an index created at the end to have no fragmentation, to use less CPU time to build, and to be slightly smaller on disk.

But at the end, all of our cores are idle. So even though it takes less CPU time, it will take more wall clock time vs creating it at the start, and then amortizing the cost over the run of the program while other cores are still doing useful work.