${layer} worker shutdown unexpectedly

[frodrigo]

While trying to import polylines I got this issue. But the import does stop, and does not do nothing after that. I could find why it fails, but my main issue here is that is does not stop it self.

$ pelias import polylines
ESC[32minfoESC[39m: [wof-pip-service:master] starting with layers neighbourhood,borough,locality,localadmin,county,macrocounty,macroregion,region,dependency,country,empire,continent,marinearea,ocean
ESC[32minfoESC[39m: [wof-pip-service:master] empire worker loaded 0 features in 0.031 seconds
ESC[32minfoESC[39m: [wof-pip-service:master] ocean worker loaded 7 features in 0.085 seconds
ESC[32minfoESC[39m: [wof-pip-service:master] dependency worker loaded 32 features in 0.217 seconds
ESC[32minfoESC[39m: [wof-pip-service:master] borough worker loaded 138 features in 0.299 seconds
ESC[32minfoESC[39m: [wof-pip-service:master] macroregion worker loaded 25 features in 0.358 seconds
ESC[32minfoESC[39m: [wof-pip-service:master] macrocounty worker loaded 23 features in 0.392 seconds
ESC[32minfoESC[39m: [wof-pip-service:master] marinearea worker loaded 305 features in 0.36 seconds
ESC[32minfoESC[39m: [wof-pip-service:master] continent worker loaded 8 features in 0.531 seconds
ESC[32minfoESC[39m: [wof-pip-service:master] country worker loaded 199 features in 5.19 seconds
ESC[32minfoESC[39m: [wof-pip-service:master] neighbourhood worker loaded 17726 features in 16.945 seconds
ESC[32minfoESC[39m: [wof-pip-service:master] region worker loaded 4268 features in 18.657 seconds
ESC[32minfoESC[39m: [wof-pip-service:master] county worker loaded 24845 features in 28.067 seconds
ESC[32minfoESC[39m: [wof-pip-service:master] localadmin worker loaded 99206 features in 56.156 seconds
ESC[32minfoESC[39m: [wof-pip-service:master] locality worker loaded 143249 features in 79.723 seconds
ESC[32minfoESC[39m: [wof-pip-service:master] PIP Service Loading Completed!!!
^CESC[31merrorESC[39m: [wof-pip-service:master] empire worker exited unexpectedly with code null, signal SIGINT
ESC[31merrorESC[39m: [wof-pip-service:master] ocean worker exited unexpectedly with code null, signal SIGINT

/code/pelias/polylines/node_modules/pelias-wof-admin-lookup/src/pip/index.js:166
        throw `${layer} worker shutdown unexpectedly`;
        ^
empire worker shutdown unexpectedly

[frodrigo]

I'm not sure what is the issue here. empire and ocean seam to be not terminated, but also written as loaded.

[missinglink]

Please post your pelias.json file.

[missinglink]

I suspect you're using imports.whosonfirst.importPlace and the place ID doesn't link to any empire or ocean records, presumably because it's a city?

[frodrigo]

I'm on importing planet https://github.com/pelias/docker/tree/master/projects/planet

I build polylines using prepare polylines.

[missinglink]

Hmm ok, thanks for the report.

cc/ @Joxit @orangejulius is it possible this is related to the PRs merged last week in the pelias/whosonfirst repo?

[missinglink]

@frodrigo in order to reproduce the bug can you please answer some questions to help the team debug it:

• what operating system are you using?
• how much RAM is available to docker?
• have you made any changes to the default planet configs you linked above?

[frodrigo]

• Debian GNU/Linux 10 (buster)
• 125 GB system
• No changes to the data or config

[missinglink]

Ok thanks, this is unexpected and likely a bug from what you've said.

You may continue your import with little impact, records will not have empire or ocean data associated in the parent hierarchy.

[frodrigo]

Thank. But I more concern by the fact that the import does not stop by itself than the data.

[missinglink]

One thing which looks unusual to me is that the signal was SIGINT which is usually sent when the user hits Ctrl+C.

I'm guessing that you didn't Ctrl+C? so if be interested to find out where that's coming from.

[frodrigo]

I hit Ctrl+C, after a day, but the same log happen after minutes.

[frodrigo]

I miss this point. The prepare polylines fails because of fatal error: runtime: out of memory. It produces a (valid?) 1 byte file extract.0sv.

Still not explain the number of features loaded on import polylines.

[missinglink]

Hmmm I thought I put a big scary warning in the log with lots of !!!!! explaining that this might happen?

[orangejulius]

It sounds like what we need to do here is add some basic sanity checking of the polyline file when the importer starts.

All the messages about admin workers loading records is normal, but for whatever reason due to the invalid nature of the polyline file, the importer hung instead of detecting the problem and quitting.

[missinglink]

Agh I see, sounds like an easy fix which would save people time debugging, I was assuming it was a PIP problem but in this case it was a polylines problem.

Also I think I'm going to change that warning into an error, if 125GB is insufficient then we may as well go ahead and say that my pbf streets command isn't suitable for large PBF files, there is already an alternative approach in the Valhalla script Kevin wrote for us.

[orangejulius]

I just tested this out and I can confirm that attempting to import an empty file (you can create one with touch, for example), will hang.

Looks to be something related to stream handling. In https://github.com/pelias/polylines/blob/master/stream/parser.js#L49 we end up calling next() without ever passing any objects down the stream, and I guess that isn't handled properly.

[missinglink]

I can have a look at this tomorrow, I think we usually use the split2 module but here we're using split, possibly related?

[missinglink]

hi @frodrigo, this should now be fixed by https://github.com/pelias/polylines/pull/247 a new docker image has been published as pelias/polylines:latest, you can pelias compose pull to update it.

if you're having issues with preparing the polylines file using Valhalla please open another issue to discuss that.

[frodrigo]

Ok for me. Thank you all.