pelias / wof-admin-lookup

Who's on First Admin Lookup for the Pelias Geocoder
https://pelias.io
MIT License
9 stars 24 forks source link

Reorganize workers to contain some data from each layer #116

Closed orangejulius closed 4 years ago

orangejulius commented 7 years ago

We have been running into build failures recently because the child processes for our admin lookup are running out of memory. A key cause of that is the way data is stored. Each admin lookup worker stores all the data from a single layer. Some layers are small, but some are very large, and those are the first to run out of memory.

I propose we reorganize the admin lookup code, so that some data from each layer is contained within each worker (and the number of workers is configurable, rather than forced to match with the number of layers). Each worker would internally keep data from each layer separate, so that queries for single layers would still work.

In paper diagram form: 20170609_155830 20170609_155839 20170609_155848 20170609_155856 20170609_155904

dianashk commented 7 years ago

@orangejulius These are some sweet paper diagrams! 📄 ✨

orangejulius commented 4 years ago

I think we've settled that we are not going to improve the architecture of the pip service at this point. The spatial service will take over instead.