Worker processes are hitting the Node.js memory limit

[orangejulius]

As the amount of data in Who's on First grows, the admin lookup workers have to store more and more data. The county layer is now so big that the admin lookup worker process for counties frequently hits the Node.js memory limit (which defaults to 1.5GB). This causes the admin lookup system to stop processing all requests as it waits for responses from the county layer worker that will never come.

This affects both the importers and the pip-service.

We should consider taking the following actions:

• [x] Increase the memory limit using the --max_old_space_size V8 flag to fix the issue temporarily
• [x] Add the ability to detect workers that have stopped for some reason and either restart them or shut down the importer/pip-service
• [x] Plan to make architectural improvements to the admin lookup system so that it's more flexible and resilient.

[dianashk]

Still need to address Add the ability to detect workers that have stopped for some reason and either restart them or shut down the importer/pip-service point. @orangejulius, please add a separate issue to track that work.

[orangejulius]

Okay, https://github.com/pelias/wof-admin-lookup/issues/138 now tracks the worker stopping issue. Everything else is done here.