Previously, these were generated four times per block.
Now they're thread-locals, so they're generated once per phase, independent of the number of blocks.
This seems to be important for PBFs from Geofabrik, which have many, small blocks vs BBBike:
Geofabrik GB extract: 26,000 blocks, 65KB/block
BBBike partial USA extract: 33 blocks, 16,000KB/block
When processing Geofabrik extracts, I noticed that my CPU was idler than I'd expect during the PBF reading phase. strace suggested that Lua initialization takes an expensive lock, blocking other readers who are also trying to initialize Lua.
PBF reading for Great Britain is a bit faster with this change:
Previously, these were generated four times per block.
Now they're thread-locals, so they're generated once per phase, independent of the number of blocks.
This seems to be important for PBFs from Geofabrik, which have many, small blocks vs BBBike:
When processing Geofabrik extracts, I noticed that my CPU was idler than I'd expect during the PBF reading phase. strace suggested that Lua initialization takes an expensive lock, blocking other readers who are also trying to initialize Lua.
PBF reading for Great Britain is a bit faster with this change:
Before:
real 1m48.685s user 21m10.017s sys 1m36.528s
After:
real 1m36.549s user 19m52.995s sys 0m54.015s