systemed / tilemaker

Make OpenStreetMap vector tiles without the stack
https://tilemaker.org/
Other
1.44k stars 229 forks source link

Re-use ifstream and OSMLuaProcessing #578

Closed cldellow closed 10 months ago

cldellow commented 10 months ago

Previously, these were generated four times per block.

Now they're thread-locals, so they're generated once per phase, independent of the number of blocks.

This seems to be important for PBFs from Geofabrik, which have many, small blocks vs BBBike:

When processing Geofabrik extracts, I noticed that my CPU was idler than I'd expect during the PBF reading phase. strace suggested that Lua initialization takes an expensive lock, blocking other readers who are also trying to initialize Lua.

PBF reading for Great Britain is a bit faster with this change:

Before:

real 1m48.685s user 21m10.017s sys 1m36.528s

After:

real 1m36.549s user 19m52.995s sys 0m54.015s

systemed commented 10 months ago

That does seem to speed up pbf reading noticeably - excellent, thank you.