systemed / tilemaker

Make OpenStreetMap vector tiles without the stack
https://tilemaker.org/
Other
1.44k stars 229 forks source link

tile index: don't block on shared lock #591

Closed cldellow closed 9 months ago

cldellow commented 9 months ago

I promise I'm running out of PRs. :)

Previously, each worker thread tried to eagerly insert their object into the index, blocking if another thread had the lock.

The index update operation can be expensive -- it potentially has to rebalance the map, and potentially has to grow + copy the value's vector.

But I think no one reads the index until the output stage, so the workers could just add their updates to a queue and move on with their other work.

That's what they now do. If, after adding the object, the queue is looking large, one worker will self-nominate to flush the queue into the map. Two queues are maintained, so that other workers aren't blocked while this flush is happening.

refactor_geometries branch for reading GB:

real 1m28.080s
user 19m26.234s
sys 0m28.319s

This branch:

real 1m20.228s
user 18m59.398s
sys 0m16.494s
cldellow commented 9 months ago

Per https://github.com/systemed/tilemaker/discussions/593#discussioncomment-7667528, the global TileIndex should just cease to exist. Will make a new PR that does that instead.