Pelias interpolation is not completing the process for san-jose-metro and portland-metro

[imran-5]

Hi, I had a Pelias North-America build previously on my system, OS: Centos 7, Ram: 32GB, CPU Cores: 12, Now I have removed everything from the server with Docker and Elastisearch and reinstalling the Pelias for small areas like san-jose-metro and portland-metro. I am facing this issue with the interpolation command, It's been more than 5 days and the command is still running, I stopped the previous process and re-run the command but it's still running and not responding, and there is no log in htop stats for Pelias processing. I have tried multiple VPS using Centos, facing the same error of no response from the interpolation command about completion.

Waiting for a reply to resolve this issue.

Thanks.

[missinglink]

Can you please let me know which version you're running?

docker-compose images

[edit] version of pelias/interpolation

[imran-5]

Thanks for the quick reply. There is no version information with the docker-compose images command.

[orangejulius]

I actually noticed this the other day as well.

We recently did some much needed cleanup of the interpolation codebase. The majority of the work was updating the CI, which shouldn't affect anything, but we did upgrade some dependencies. I think it might be this PR that broke things: https://github.com/pelias/interpolation/pull/268

[imran-5]

@orangejulius Thanks for the reply, But I am not sure where to track this error. I ran the command for docker images and it's using the 11 days old image for interpolation.

And the last update on the docker hub is 12 days old.

[orangejulius]

@imran-5 No problem, I was just acknowledging that this is probably a legitimate bug in the interpolation import process. We don't have a fix yet, and I definitely don't expect you to track it down. Although of course I won't stop you from diving into the interpolation code :)

If you want a workaround for now, you can probably use an older Docker image. Try updating docker-compose.yml with this image for interpolation: pelias/interpolation:master-2021-09-29-9222a8b33436cc854e44441c2b2d76d718d8ef60

[missinglink]

It could be caused by this: https://github.com/pelias/interpolation/issues/272

There is this new option of 'unsafe mode' that I contributed to the better-sqlite module, without that it could be that attempting to read and write to the same database table concurrently results in a lock which causes an infinite block on the reader.

Just speculation.. would need to test it out.

[jlowe000]

I saw this behaviour in the vertices.js logic and when I ran this within the docker image itself, ./interpolate vertices /data/interpolation/address.db /data/interpolation/street.db I got the following error. TypeError: This database connection is busy executing a query and from that it seems to "loop". Specifically is looking at /code/pelias/interpolation/stream/address/import.js:36:8

[orangejulius]

Thanks @jlowe000, that tracks with the idea that we aren't properly enabling the 'unsafe mode' that @missinglink mentioned.

I looked up the new syntax and opened https://github.com/pelias/interpolation/pull/273 to fix things. It appears to work for me. If everyone could test it out and report back that would help!

[jlowe000]

Hi @orangejulius - I've done a quick test with the full pelias prepare all and it works with the vertices (where previously it was hanging with 0 records in 0 secs). I'm in the process doing the import of the prepared data and then I'll know for sure. But I am confident. Thanks for the quick turnaround.

[jlowe000]

Hi @orangejulius - I've tested the import and it looks to be good. I appreciate what you (and @missinglink) have done because I recently did a pull of the latest code and I wasn't sure if what was happening (because it was working a few months ago).

[orangejulius]

Hi folks, we've merged the fix from https://github.com/pelias/interpolation/pull/273 so a pelias compose pull should get everything working again. Thanks for the reports!