Closed orangejulius closed 4 years ago
@orangejulius These are some sweet paper diagrams! 📄 ✨
I think we've settled that we are not going to improve the architecture of the pip service at this point. The spatial service will take over instead.
We have been running into build failures recently because the child processes for our admin lookup are running out of memory. A key cause of that is the way data is stored. Each admin lookup worker stores all the data from a single layer. Some layers are small, but some are very large, and those are the first to run out of memory.
I propose we reorganize the admin lookup code, so that some data from each layer is contained within each worker (and the number of workers is configurable, rather than forced to match with the number of layers). Each worker would internally keep data from each layer separate, so that queries for single layers would still work.
In paper diagram form: