Currently shapefile reading is entirely single-threaded. This PR shunts the geometry processing into separate threads.
The Lua attribute_function doesn't do much, only basic tag remapping. The expensive bit is the geometry assembly, which includes our old friends geom::intersection and make_valid. So we can get away with putting a single mutex around the Lua access rather than creating a Lua state for each thread.
Total processing time is reduced by 6m30 for North America. (The North America .pbf crosses the 180° meridian so it ends up pulling in shapefiles for -180° to 180°.)
This also moves some geometry error output to verbose mode.
The shapefile reader code is very old and needs tidying up and moving into a ShpReader class, but this will do for now. Ideally we should refactor the StoreShapefileGeometry calls out of processShapeGeometry and put them in the calling lambda, which would make more sense conceptually and mean we didn't have to pass so many parameters around. But that will require a little bit of refactoring work on multilinestrings.
Currently shapefile reading is entirely single-threaded. This PR shunts the geometry processing into separate threads.
The Lua
attribute_functiondoesn't do much, only basic tag remapping. The expensive bit is the geometry assembly, which includes our old friendsgeom::intersectionandmake_valid. So we can get away with putting a single mutex around the Lua access rather than creating a Lua state for each thread.Total processing time is reduced by 6m30 for North America. (The North America .pbf crosses the 180° meridian so it ends up pulling in shapefiles for -180° to 180°.)
This also moves some geometry error output to verbose mode.
The shapefile reader code is very old and needs tidying up and moving into a
ShpReaderclass, but this will do for now. Ideally we should refactor theStoreShapefileGeometrycalls out of processShapeGeometry and put them in the calling lambda, which would make more sense conceptually and mean we didn't have to pass so many parameters around. But that will require a little bit of refactoring work on multilinestrings.