Using pelias and getting a wof error

[LDuderino-zz]

Hi, We have been using pelias and as I was uploading the latest data from openstreetmaps, I encountered the following message -- which seems to indicate that something has gone wrong after the integration with wof. We were using adminLookup when it used to work. Can anyone point me to what could be the problem and how to fix it? Is it a matter of adding some parameter while starting the import? Or, something more involved?

Thanks.

`nohup: ignoring input /opt/pelias/osm/releases/20160210163303/stream/adminLookup.js:45 throw new Error(message);

    ^

Error: Admin lookup is enabled but no url is specified in your config at imports.adminLookup.url. Make sure a URL pointing to a Who's on First point in polygon server (https://github.com/whosonfirst/go-whosonfirst-pip) is set in your config

at Object.createStream [as adminLookup] (/opt/pelias/osm/releases/20160210163303/stream/adminLookup.js:45:9)

at Object.streams.import (/opt/pelias/osm/releases/20160210163303/index.js:37:20)

at Object.<anonymous> (/opt/pelias/osm/releases/20160210163303/index.js:50:17)

at Module._compile (module.js:460:26)

at Object.Module._extensions..js (module.js:478:10)

at Module.load (module.js:355:32)

at Function.Module._load (module.js:310:12)

at Function.Module.runMain (module.js:501:10)

at startup (node.js:129:16)

at node.js:814:3

Command exited with non-zero status 1

0.85user 0.35system 0:03.20elapsed 37%CPU (0avgtext+0avgdata 76184maxresident)k 23032inputs+0outputs (3major+19260minor)pagefaults 0swaps`

[orangejulius]

Hey @LDuderino, We were briefly using a separate service to do the point-in-polygon intersection required to gather country, city, neighborhood, and other information for admin lookup. However at the moment we've switched back to using a system that runs automatically without additional configuration needed. It should be available if you update to the latest master of the openstreetmap repo. Because Whosonfirst(WOF) is a larger, newer dataset than quattroshapes, it will require more RAM to do admin lookup, about 3.5GB last time I checked, so be aware of that. Also be sure to run NPM update since the changes are really only in packages that the importers depend on.

In general, dding WOF to Pelias has caused the master branches to be a bit more unstable than they have been in the past, although it should be settling down in the next couple weeks. If you want to keep things more stable until then, feel free to stay on the production branches, which haven't yet received any of those changes, and they should behave exactly as before.

Let me know if you have more questions about our importers, branches, WOF, etc.

[orangejulius]

Oops, actually this is a bug cause by some "helpful" error messages we added, but aren't needed anymore, can you try out the PR I just opened up to fix this?

[LDuderino-zz]

Cool Thanks @orangejulius . Will give it a shot.

[LDuderino-zz]

Thanks that patch worked!

One quick question: Has the format changed? I do not see admin0, admin1 fields anymore. And I do have adminLookup turned on for osm.

[orangejulius]

No format changes, in fact there is another bug that caused the admin fields to not show up. It's fixed in pelias/wof-admin-lookup#19 which will be making its way through soon.

[LDuderino-zz]

I still do not see any admin0, admin1 fields after I restarted with your latest changes.

[orangejulius]

Hey @LDuderino, The fix is in 8c6aba1 (currently the HEAD of the master branch). If you're already on that commit, try rerunning npm install or removing the node_modules dir. If that doesn't work you've probably found another bug, and we'd love to know about it.

[LDuderino-zz]

Hi @orangejulius . I am on the right commit, (i.e., I do see it on the top) and I did do npm install. But I had not removed the node_modules directory. After removing that directory when I restart the process to import, I get this error now:

nohup: ignoring input module.js:338 throw err; ^ Error: Cannot find module 'through2-spy' at Function.Module._resolveFilename (module.js:336:15) at Function.Module._load (module.js:278:25) at Module.require (module.js:365:17) at require (module.js:384:17) at Object.<anonymous> (/opt/pelias/osm/index.js:7:11) at Module._compile (module.js:460:26) at Object.Module._extensions..js (module.js:478:10) at Module.load (module.js:355:32) at Function.Module._load (module.js:310:12) at Function.Module.runMain (module.js:501:10) Command exited with non-zero status 1 0.12user 0.00system 0:00.12elapsed 93%CPU (0avgtext+0avgdata 19924maxresident)k 8inputs+0outputs (0major+2754minor)pagefaults 0swaps

[orangejulius]

You have to re-run npm install after clearing out node_modules. :P Sorry that wasn't clear. It's a bit like rebooting a windows machine in that it seems to fix many problems and no one really knows why.

[LDuderino-zz]

Thanks @orangejulius. Yes, now the error is gone. However, I did a search on Elasticsearch and I noticed that the data still does not contain admin0, admin1,... fields. Maybe, they come in later and are region specific(?) and I have yet to see them? I will leave it running and see if I spot it after a day's worth of loading. But let me know if this is problematic enough and if you think there is something else going on or if there is another bug or improper config in my setup.

Here is the gist of the json of 10 random records that were loaded:

[LDuderino-zz]

Oh, and I had deleted the index before reloading. Just FYI.

[LDuderino-zz]

@orangejulius Any clues/ideas on why admin0, admin1,... fields are not populated? It has something to do with the recent changes about adminLookup me thinks...

[LDuderino-zz]

Any word on diagnosis/fix ETA for this? @dianashk @orangejulius ? Thanks.

[orangejulius]

Hey, sorry about the delay, I'm at a conference this week so I have not been good at responding to issues.

We just finished a successful build in our dev environment that included admin fields for OSM just fine. This only says that it is possible to get things to work, not that they will always work, however.

My most basic suggestions to ensure its not something simple are:

1. Update to the latest OSM importer code
2. rm -rf node_modules && npm install to ensure all the module dependencies are up to date
3. Make sure adminLookup: true is in the OSM importer section of your pelias config file
4. You should see at least 3GB of memory usage across something like 8 node processes during the import. If you only see one process, the admin lookup is simply not running. Probably its a pelias.config issue, but it could also be a bug where the importer is not running when it should.

Let me know if any of that helps or leads to clues that can help us debug further On Feb 18, 2016 8:26 AM, "NT" notifications@github.com wrote:

Any word on diagnosis/fix ETA for this? @dianashk https://github.com/dianashk @orangejulius https://github.com/orangejulius ? Thanks.

— Reply to this email directly or view it on GitHub https://github.com/pelias/whosonfirst/issues/38#issuecomment-185801406.

[dianashk]

@LDuderino, have you downloaded the Who's On First data? In order to perform the admin lookup, we now need all the WOF data to be on the importer machine.

You can perform the download using these instructions, from our new whosonfirst importer

Once you have that somewhere on the machine, you will also need to specify the data directory path in the Pelias config file, like so:

{
  "imports": {
    "whosonfirst": {
      "datapath": "/mnt/data/whosonfirst-data/"
    },
    "openstreetmap": {
      "leveldbpath": "/tmp",
      "datapath": "/mnt/data/openstreetmap/",
      "adminLookup": true,
      "import": [
        {
          "filename": "vancouver_canada.osm.pbf"
        }
      ]
    }
  }
}

Once all of that is in place, you should start seeing admin fields populate. Let us know how that works out for you.

[LDuderino-zz]

Thanks @orangejulius @dianashk for the helpful tips. I will start afresh and see where it goes.

[orangejulius]

One more tip now that I am on a real computer: enable a higher level of logging in your pelias config. a section like this will really help. you should see lots of admin-lookup related messages at the start of an import.

    "logger": {
        "level": "debug",
        "timestamp": true
    },  

[LDuderino-zz]

Thanks @orangejulius . I think I am missing the WOF data as @dianashk pointed out. I have a screenshot of the ps -ef | grep -i node from when it was running and I do see 8 processes. All other things in your checklist had been taken care of AFAICT.

pelias    6261   944  0 Feb13 ?        00:00:21 node index.js
pelias    7814  6261  0 00:47 ?        00:00:06 /usr/local/nodejs-binary-0.12.2/bin/node --debug-port=5883 /opt/pelias/pelias-api/index.js
pelias    7823  6261  0 00:47 ?        00:00:05 /usr/local/nodejs-binary-0.12.2/bin/node --debug-port=5884 /opt/pelias/pelias-api/index.js
pelias    7832  6261  0 00:47 ?        00:00:05 /usr/local/nodejs-binary-0.12.2/bin/node --debug-port=5885 /opt/pelias/pelias-api/index.js
pelias    7833  6261  0 00:47 ?        00:00:05 /usr/local/nodejs-binary-0.12.2/bin/node --debug-port=5886 /opt/pelias/pelias-api/index.js
pelias    7834  6261  0 00:47 ?        00:00:05 /usr/local/nodejs-binary-0.12.2/bin/node --debug-port=5887 /opt/pelias/pelias-api/index.js
pelias    7835  6261  0 00:47 ?        00:00:05 /usr/local/nodejs-binary-0.12.2/bin/node --debug-port=5888 /opt/pelias/pelias-api/index.js
pelias    7836  6261  0 00:47 ?        00:00:05 /usr/local/nodejs-binary-0.12.2/bin/node --debug-port=5889 /opt/pelias/pelias-api/index.js
pelias    7837  6261  0 00:47 ?        00:00:05 /usr/local/nodejs-binary-0.12.2/bin/node --debug-port=5890 /opt/pelias/pelias-api/index.js
pelias    8273  8249  0 22:08 pts/0    00:00:00 grep --color=auto node

I will let you know soon-ish how it goes.

[orangejulius]

And finally, here's a full OSM Elasticsearch document from our dev index to see what a document "should" look like. Elasticsearch gave me this at random, so if you want to see a particular OSM node/way for any reason let me know.

 {
            "_index": "pelias",
            "_type": "address",
            "_id": "node:2629101305",
            "_score": 2.1911113,
            "_source": {
               "parent": {
                  "county_abbr": [
                     null
                  ],
                  "localadmin_abbr": [],
                  "country": [
                     "Germany"
                  ],
                  "neighbourhood_id": [],
                  "region_id": [
                     "85682541"
                  ],
                  "county": [
                     "Suhl"
                  ],
                  "locality": [
                     "Suhl"
                  ],
                  "macroregion": [],
                  "localadmin": [],
                  "locality_abbr": [
                     null
                  ],
                  "macrocounty_abbr": [],
                  "macrocounty": [],
                  "county_id": [
                     "102063649"
                  ],
                  "locality_id": [
                     "101841163"
                  ],
                  "neighbourhood_abbr": [],
                  "neighbourhood": [],
                  "macroregion_abbr": [],
                  "region_abbr": [
                     null
                  ],
                  "localadmin_id": [],
                  "country_abbr": [
                     null
                  ],
                  "region": [
                     "Thüringen"
                  ],
                  "macrocounty_id": [],
                  "country_id": [
                     "85633111"
                  ],
                  "macroregion_id": []
               },
               "address": {
                  "zip": "98527",
                  "number": "5 a",
                  "street": "Am Holunderbusch"
               },
               "locality": "Suhl",
               "source": "openstreetmap",
               "layer": "address",
               "center_point": {
                  "lon": 10.699234,
                  "lat": 50.606387
               },
               "name": {
                  "default": "5 a Am Holunderbusch"
               },
               "admin1": "Thüringen",
               "alpha3": "DEU",
               "admin2": "Suhl",
               "source_id": "node:2629101305",
               "admin0": "Germany",
               "category": []
            }

[orangejulius]

Ahhh, looking at your PS output, I see that the first process is the indexing process, and the others belong to the API server. Here's what I see during an import:

spectre256@laguna ~/repos/pelias/openaddresses $ ps aux | grep node
spectre+ 14625 17.6  1.9 742928 78972 pts/4    Sl+  18:40   0:00 node import.js --admin-values
spectre+ 14634 66.7  7.6 996500 302124 pts/4   Rl+  18:40   0:02 /usr/bin/node /home/spectre256/repos/pelias/openaddresses/node_modules/pelias-wof-admin-lookup/node_modules/pelias-wof-pip-service/src/worker
spectre+ 14639 45.0  1.5 746256 63112 pts/4    Dl+  18:40   0:01 /usr/bin/node /home/spectre256/repos/pelias/openaddresses/node_modules/pelias-wof-admin-lookup/node_modules/pelias-wof-pip-service/src/worker
spectre+ 14640 20.2  1.4 741744 58164 pts/4    Sl+  18:40   0:00 /usr/bin/node /home/spectre256/repos/pelias/openaddresses/node_modules/pelias-wof-admin-lookup/node_modules/pelias-wof-pip-service/src/worker
spectre+ 14645 15.2  1.1 727408 44004 pts/4    Sl+  18:40   0:00 /usr/bin/node /home/spectre256/repos/pelias/openaddresses/node_modules/pelias-wof-admin-lookup/node_modules/pelias-wof-pip-service/src/worker
spectre+ 14650 41.5  1.1 731516 47100 pts/4    Dl+  18:40   0:01 /usr/bin/node /home/spectre256/repos/pelias/openaddresses/node_modules/pelias-wof-admin-lookup/node_modules/pelias-wof-pip-service/src/worker
spectre+ 14655 39.5  1.6 749212 65012 pts/4    Rl+  18:40   0:01 /usr/bin/node /home/spectre256/repos/pelias/openaddresses/node_modules/pelias-wof-admin-lookup/node_modules/pelias-wof-pip-service/src/worker
spectre+ 14660 30.2  1.4 737880 59024 pts/4    Sl+  18:40   0:01 /usr/bin/node /home/spectre256/repos/pelias/openaddresses/node_modules/pelias-wof-admin-lookup/node_modules/pelias-wof-pip-service/src/worker
spectre+ 14665 67.5  3.1 801764 124024 pts/4   Rl+  18:40   0:02 /usr/bin/node /home/spectre256/repos/pelias/openaddresses/node_modules/pelias-wof-admin-lookup/node_modules/pelias-wof-pip-service/src/worker
spectre+ 14674 42.0  1.5 744804 60708 pts/4    Rl+  18:40   0:01 /usr/bin/node /home/spectre256/repos/pelias/openaddresses/node_modules/pelias-wof-admin-lookup/node_modules/pelias-wof-pip-service/src/worker
spectre+ 14675 45.0  1.6 748424 66304 pts/4    Rl+  18:40   0:01 /usr/bin/node /home/spectre256/repos/pelias/openaddresses/node_modules/pelias-wof-admin-lookup/node_modules/pelias-wof-pip-service/src/worker

[LDuderino-zz]

Interesting. I will keep an eye for these processes next time I load. Probably tomorrow.

[LDuderino-zz]

@dianashk I was unable to download Who's on First data. I tried two options.

Option 1

I git cloned the repo using git clone https://github.com/pelias/whosonfirst.git

The following is the ls -l output after doing a cd whosonfirst

total 28
-rw-rw-r-- 1 pelias pelias 2124 Feb 19 23:51 download_data.js
-rw-rw-r-- 1 pelias pelias 1722 Feb 19 23:51 import.js
-rw-rw-r-- 1 pelias pelias 1776 Feb 19 23:53 npm-debug.log
-rw-rw-r-- 1 pelias pelias 1355 Feb 19 23:51 package.json
-rw-rw-r-- 1 pelias pelias 1249 Feb 19 23:51 README.md
drwxrwxr-x 2 pelias pelias 4096 Feb 19 23:51 src
drwxrwxr-x 2 pelias pelias 4096 Feb 19 23:51 test

When I try to do npm run download_data per the instructions, I get the following error:

npm ERR! Linux 3.19.0-25-generic
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "run" "download_data"
npm ERR! node v0.12.2
npm ERR! npm  v2.7.4
npm ERR! missing script: download_data
npm ERR! 
npm ERR! If you need help, you may report this error at:
npm ERR!     
npm ERR! Please include the following file with any support request:
npm ERR!     /opt/pelias/whosonfirst/npm-debug.log
pelias@pelias:/opt/pelias/whosonfirst$ npm start
> whosonfirst@1.0.0 start /opt/pelias/whosonfirst
> node import.js
module.js:338
    throw err;
          ^
Error: Cannot find module 'pelias-config'
    at Function.Module._resolveFilename (module.js:336:15)
    at Function.Module._load (module.js:278:25)
    at Module.require (module.js:365:17)
    at require (module.js:384:17)
    at Object. (/opt/pelias/whosonfirst/import.js:1:82)
    at Module._compile (module.js:460:26)
    at Object.Module._extensions..js (module.js:478:10)
    at Module.load (module.js:355:32)
    at Function.Module._load (module.js:310:12)
    at Function.Module.runMain (module.js:501:10)
npm ERR! Linux 3.19.0-25-generic
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "start"
npm ERR! node v0.12.2
npm ERR! npm  v2.7.4
npm ERR! code ELIFECYCLE
npm ERR! whosonfirst@1.0.0 start: `node import.js`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the whosonfirst@1.0.0 start script 'node import.js'.
npm ERR! This is most likely a problem with the whosonfirst package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node import.js
npm ERR! You can get their info via:
npm ERR!     npm owner ls whosonfirst
npm ERR! There is likely additional logging output above.
npm ERR! Please include the following file with any support request:
npm ERR!     /opt/pelias/whosonfirst/npm-debug.log

Option 2

Per instructions, I tried git clone https://github.com/whosonfirst/whosonfirst-data.git and I got the error:

fatal: could not create work tree dir 'whosonfirst-data'.: Permission denied

I am not a node expert -- all I want is to import the planet data with admin fields. But you mentioned we need WOF data and now I am stuck because I cannot download WOF data. This used to work just fine before this WOF was there. Can you please help and fix the README on WOF because that stuff does not work!!

Thanks.

[trescube]

I've corrected README.md, it should have read npm run download, not npm run download_data.

Regards, Stephen

On Fri, Feb 19, 2016 at 4:11 PM, NT notifications@github.com wrote:

@dianashk https://github.com/dianashk I was unable to download Who's on First data. I tried two options. Option 1

I git cloned the repo using git clone https://github.com/pelias/whosonfirst.git

The following is the ls -l output after doing a cd whosonfirst

total 28 -rw-rw-r-- 1 pelias pelias 2124 Feb 19 23:51 download_data.js -rw-rw-r-- 1 pelias pelias 1722 Feb 19 23:51 import.js -rw-rw-r-- 1 pelias pelias 1776 Feb 19 23:53 npm-debug.log -rw-rw-r-- 1 pelias pelias 1355 Feb 19 23:51 package.json -rw-rw-r-- 1 pelias pelias 1249 Feb 19 23:51 README.md drwxrwxr-x 2 pelias pelias 4096 Feb 19 23:51 src drwxrwxr-x 2 pelias pelias 4096 Feb 19 23:51 test

When I try to do npm run download_data per the instructions https://github.com/pelias/whosonfirst/blob/master/README.md#data, I get the following error:

npm ERR! Linux 3.19.0-25-generic npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "run" "download_data" npm ERR! node v0.12.2 npm ERR! npm v2.7.4

npm ERR! missing script: download_data npm ERR! npm ERR! If you need help, you may report this error at: npm ERR!

npm ERR! Please include the following file with any support request: npm ERR! /opt/pelias/whosonfirst/npm-debug.log pelias@pelias:/opt/pelias/whosonfirst$ npm start

whosonfirst@1.0.0 start /opt/pelias/whosonfirst node import.js

module.js:338 throw err; ^ Error: Cannot find module 'pelias-config' at Function.Module._resolveFilename (module.js:336:15) at Function.Module._load (module.js:278:25) at Module.require (module.js:365:17) at require (module.js:384:17) at Object. (/opt/pelias/whosonfirst/import.js:1:82) at Module._compile (module.js:460:26) at Object.Module._extensions..js (module.js:478:10) at Module.load (module.js:355:32) at Function.Module._load (module.js:310:12) at Function.Module.runMain (module.js:501:10)

npm ERR! Linux 3.19.0-25-generic npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "start" npm ERR! node v0.12.2 npm ERR! npm v2.7.4 npm ERR! code ELIFECYCLE npm ERR! whosonfirst@1.0.0 start: node import.js npm ERR! Exit status 1 npm ERR! npm ERR! Failed at the whosonfirst@1.0.0 start script 'node import.js'. npm ERR! This is most likely a problem with the whosonfirst package, npm ERR! not with npm itself. npm ERR! Tell the author that this fails on your system: npm ERR! node import.js npm ERR! You can get their info via: npm ERR! npm owner ls whosonfirst npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request: npm ERR! /opt/pelias/whosonfirst/npm-debug.log

Option 2

Per instructions https://github.com/pelias/whosonfirst/blob/master/README.md#data, I tried git clone https://github.com/whosonfirst/whosonfirst-data.git and I got the error:

fatal: could not create work tree dir 'whosonfirst-data'.: Permission denied

I am not a node expert -- all I want is to import the planet data with admin fields. But you mentioned we need WOF data and now I am stuck because I cannot download WOF data. This used to work just fine before this WOF was there. Can you please help and fix the README on WOF because that stuff does not work!!

Thanks.

— Reply to this email directly or view it on GitHub https://github.com/pelias/whosonfirst/issues/38#issuecomment-186462601.

[LDuderino-zz]

@trescube I get a different error now:

> whosonfirst@1.0.0 download /opt/pelias/whosonfirst
> node download_data.js
module.js:338
    throw err;
          ^
Error: Cannot find module 'unbzip2-stream'
    at Function.Module._resolveFilename (module.js:336:15)
    at Function.Module._load (module.js:278:25)
    at Module.require (module.js:365:17)
    at require (module.js:384:17)
    at Object. (/opt/pelias/whosonfirst/download_data.js:1:73)
    at Module._compile (module.js:460:26)
    at Object.Module._extensions..js (module.js:478:10)
    at Module.load (module.js:355:32)
    at Function.Module._load (module.js:310:12)
    at Function.Module.runMain (module.js:501:10)
npm ERR! Linux 3.19.0-25-generic
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "run" "download"
npm ERR! node v0.12.2
npm ERR! npm  v2.7.4
npm ERR! code ELIFECYCLE
npm ERR! whosonfirst@1.0.0 download: `node download_data.js`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the whosonfirst@1.0.0 download script 'node download_data.js'.
npm ERR! This is most likely a problem with the whosonfirst package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node download_data.js
npm ERR! You can get their info via:
npm ERR!     npm owner ls whosonfirst
npm ERR! There is likely additional logging output above.
npm ERR! Please include the following file with any support request:
npm ERR!     /opt/pelias/whosonfirst/npm-debug.log

Do I have to do npm start first? Is there an easier way to get the data? So I can move along?

[trescube]

Okay, I'll take a look right now.

On Fri, Feb 19, 2016 at 4:36 PM, NT notifications@github.com wrote:

@trescube https://github.com/trescube I get a different error now:

whosonfirst@1.0.0 download /opt/pelias/whosonfirst node download_data.js

module.js:338 throw err; ^ Error: Cannot find module 'unbzip2-stream' at Function.Module._resolveFilename (module.js:336:15) at Function.Module._load (module.js:278:25) at Module.require (module.js:365:17) at require (module.js:384:17) at Object. (/opt/pelias/whosonfirst/download_data.js:1:73) at Module._compile (module.js:460:26) at Object.Module._extensions..js (module.js:478:10) at Module.load (module.js:355:32) at Function.Module._load (module.js:310:12) at Function.Module.runMain (module.js:501:10)

npm ERR! Linux 3.19.0-25-generic npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "run" "download" npm ERR! node v0.12.2 npm ERR! npm v2.7.4 npm ERR! code ELIFECYCLE npm ERR! whosonfirst@1.0.0 download: node download_data.js npm ERR! Exit status 1 npm ERR! npm ERR! Failed at the whosonfirst@1.0.0 download script 'node download_data.js'. npm ERR! This is most likely a problem with the whosonfirst package, npm ERR! not with npm itself. npm ERR! Tell the author that this fails on your system: npm ERR! node download_data.js npm ERR! You can get their info via: npm ERR! npm owner ls whosonfirst npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request: npm ERR! /opt/pelias/whosonfirst/npm-debug.log

Do I have to do npm start first? Is there an easier way to get the data? So I can move along?

— Reply to this email directly or view it on GitHub https://github.com/pelias/whosonfirst/issues/38#issuecomment-186467423.

[trescube]

Oh, sorry, I'm new-ish to node and still learning what should be specified, but you have to run npm i first to install all the node dependencies. I'll fix the documentation.

On Fri, Feb 19, 2016 at 4:37 PM, Stephen Hess stephen.k.hess@gmail.com wrote:

Okay, I'll take a look right now.

On Fri, Feb 19, 2016 at 4:36 PM, NT notifications@github.com wrote:

@trescube https://github.com/trescube I get a different error now:

whosonfirst@1.0.0 download /opt/pelias/whosonfirst node download_data.js

module.js:338 throw err; ^ Error: Cannot find module 'unbzip2-stream' at Function.Module._resolveFilename (module.js:336:15) at Function.Module._load (module.js:278:25) at Module.require (module.js:365:17) at require (module.js:384:17) at Object. (/opt/pelias/whosonfirst/download_data.js:1:73) at Module._compile (module.js:460:26) at Object.Module._extensions..js (module.js:478:10) at Module.load (module.js:355:32) at Function.Module._load (module.js:310:12) at Function.Module.runMain (module.js:501:10)

npm ERR! Linux 3.19.0-25-generic npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "run" "download" npm ERR! node v0.12.2 npm ERR! npm v2.7.4 npm ERR! code ELIFECYCLE npm ERR! whosonfirst@1.0.0 download: node download_data.js npm ERR! Exit status 1 npm ERR! npm ERR! Failed at the whosonfirst@1.0.0 download script 'node download_data.js'. npm ERR! This is most likely a problem with the whosonfirst package, npm ERR! not with npm itself. npm ERR! Tell the author that this fails on your system: npm ERR! node download_data.js npm ERR! You can get their info via: npm ERR! npm owner ls whosonfirst npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request: npm ERR! /opt/pelias/whosonfirst/npm-debug.log

Do I have to do npm start first? Is there an easier way to get the data? So I can move along?

— Reply to this email directly or view it on GitHub https://github.com/pelias/whosonfirst/issues/38#issuecomment-186467423.

[LDuderino-zz]

@trescube No worries. I am not familiar with node either. I did npm install followed by npm run download and this time it looked like it was doing something but ultimately failed(progress, but not enough):

pelias@pelias:/opt/pelias/whosonfirst$ npm run download
> whosonfirst@1.0.0 download /opt/pelias/whosonfirst
> node download_data.js
starting continent
starting country
starting county
starting dependency
starting disputed
starting localadmin
starting locality
starting macrocounty
starting macroregion
starting neighbourhood
starting region
done disputed
done dependency
done macrocounty
done continent
/opt/pelias/whosonfirst/node_modules/unbzip2-stream/index.js:62
                throw e;
                      ^
TypeError: Cannot read property '0' of undefined
    at f (/opt/pelias/whosonfirst/node_modules/unbzip2-stream/lib/bit_iterator.js:24:34)
    at Object.bzip2.decompress (/opt/pelias/whosonfirst/node_modules/unbzip2-stream/lib/bzip2.js:277:28)
    at decompressBlock (/opt/pelias/whosonfirst/node_modules/unbzip2-stream/index.js:29:28)
    at decompressAndQueue (/opt/pelias/whosonfirst/node_modules/unbzip2-stream/index.js:46:20)
    at Stream.end (/opt/pelias/whosonfirst/node_modules/unbzip2-stream/index.js:86:23)
    at _end (/opt/pelias/whosonfirst/node_modules/unbzip2-stream/node_modules/through/index.js:65:9)
    at Stream.stream.end (/opt/pelias/whosonfirst/node_modules/unbzip2-stream/node_modules/through/index.js:74:5)
    at IncomingMessage.onend (_stream_readable.js:505:10)
    at IncomingMessage.g (events.js:199:16)
    at IncomingMessage.emit (events.js:129:20)
npm ERR! Linux 3.19.0-25-generic
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "run" "download"
npm ERR! node v0.12.2
npm ERR! npm  v2.7.4
npm ERR! code ELIFECYCLE
npm ERR! whosonfirst@1.0.0 download: `node download_data.js`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the whosonfirst@1.0.0 download script 'node download_data.js'.
npm ERR! This is most likely a problem with the whosonfirst package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node download_data.js
npm ERR! You can get their info via:
npm ERR!     npm owner ls whosonfirst
npm ERR! There is likely additional logging output above.
npm ERR! Please include the following file with any support request:
npm ERR!     /opt/pelias/whosonfirst/npm-debug.log

[trescube]

Okay, I'll look into this ASAP, but it might be a little bit. Our office WIFI is going down for maintenance in a few but I'm stuck in San Francisco due to canceled flights so I'll be able to dedicate quite a bit of time once WIFI comes back.

On Fri, Feb 19, 2016 at 4:46 PM, NT notifications@github.com wrote:

@trescube https://github.com/trescube No worries. I am not familiar with node either. I did npm install followed by npm run download and this time it looked like it was doing something but ultimately failed(progress, but not enough):

pelias@pelias:/opt/pelias/whosonfirst$ npm run download

whosonfirst@1.0.0 download /opt/pelias/whosonfirst node download_data.js

starting continent starting country starting county starting dependency starting disputed starting localadmin starting locality starting macrocounty starting macroregion starting neighbourhood starting region done disputed done dependency done macrocounty done continent /opt/pelias/whosonfirst/node_modules/unbzip2-stream/index.js:62 throw e; ^ TypeError: Cannot read property '0' of undefined at f (/opt/pelias/whosonfirst/node_modules/unbzip2-stream/lib/bit_iterator.js:24:34) at Object.bzip2.decompress (/opt/pelias/whosonfirst/node_modules/unbzip2-stream/lib/bzip2.js:277:28) at decompressBlock (/opt/pelias/whosonfirst/node_modules/unbzip2-stream/index.js:29:28) at decompressAndQueue (/opt/pelias/whosonfirst/node_modules/unbzip2-stream/index.js:46:20) at Stream.end (/opt/pelias/whosonfirst/node_modules/unbzip2-stream/index.js:86:23) at _end (/opt/pelias/whosonfirst/node_modules/unbzip2-stream/node_modules/through/index.js:65:9) at Stream.stream.end (/opt/pelias/whosonfirst/node_modules/unbzip2-stream/node_modules/through/index.js:74:5) at IncomingMessage.onend (_stream_readable.js:505:10) at IncomingMessage.g (events.js:199:16) at IncomingMessage.emit (events.js:129:20)

npm ERR! Linux 3.19.0-25-generic npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "run" "download" npm ERR! node v0.12.2 npm ERR! npm v2.7.4 npm ERR! code ELIFECYCLE npm ERR! whosonfirst@1.0.0 download: node download_data.js npm ERR! Exit status 1 npm ERR! npm ERR! Failed at the whosonfirst@1.0.0 download script 'node download_data.js'. npm ERR! This is most likely a problem with the whosonfirst package, npm ERR! not with npm itself. npm ERR! Tell the author that this fails on your system: npm ERR! node download_data.js npm ERR! You can get their info via: npm ERR! npm owner ls whosonfirst npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request: npm ERR! /opt/pelias/whosonfirst/npm-debug.log

— Reply to this email directly or view it on GitHub https://github.com/pelias/whosonfirst/issues/38#issuecomment-186469669.

[LDuderino-zz]

@trescube Sure thing. I am heading out for the weekend too. I will keep checking. If this starts working for me I would like to kick off the planet import over the weekend - which takes a couple of days to finish anyway. Thanks for your help!

[LDuderino-zz]

@trescube By the way, how big is the WOF data? Is it possible for you guys to send it to me while you work on fixing this repo?

[trescube]

The bundles that the downloader gets are here:

https://whosonfirst.mapzen.com/bundles/

You'll need the following bundles:

continent country county dependency disputed localadmin locality macrocounty macroregion neighbourhood region

I think it's around 5gb uncompressed. The whosonfirst importer expects data to be in the directory configuration:

download_dir/meta download_dir/data

Move the .csv files from each bundle into download_dir/meta. Move/copy the contents of each bundle's data directory into download_dir/data. Once you're done with that, point your pelias config file to where you downloaded the data (download_dir) as specified in the README.

On Fri, Feb 19, 2016 at 4:53 PM, NT notifications@github.com wrote:

@trescube https://github.com/trescube By the way, how big is the WOF data? Is it possible for you guys to send it to me while you work on fixing this repo?

— Reply to this email directly or view it on GitHub https://github.com/pelias/whosonfirst/issues/38#issuecomment-186470914.

[LDuderino-zz]

@trescube Just a quick update. I was able to load the data. I did rm -rf node-modules, followed by npm install and npm start (though npm start gave errors). I decided to move on fearlessly regardless. After that I did npm run download and it finished this time with all 11 bundles' worth of data. So, I still think you should investigate why this was giving errors but I think I am ok for now.

[LDuderino-zz]

@orangejulius I tried everything that you and @dianashk have suggested. If you scroll up you will see the teething troubles I had with the loading wof data but that is done and I updated the pelias config file to add the imports.whosonfirst.datapath to the appropriate directory. Kicked off the import and still no sign of admin0, admin1 fields.

ps -ef | grep node shows the following. No adminlookup processes.

pelias     959   951  0 Feb19 ?        00:00:01 node index.js
pelias    1178   959  0 Feb19 ?        00:00:02 /usr/local/nodejs-binary-0.12.2/bin/node --debug-port=5859 /opt/pelias/pelias-api/index.js
pelias    1179   959  0 Feb19 ?        00:00:02 /usr/local/nodejs-binary-0.12.2/bin/node --debug-port=5860 /opt/pelias/pelias-api/index.js
pelias    1180   959  0 Feb19 ?        00:00:02 /usr/local/nodejs-binary-0.12.2/bin/node --debug-port=5861 /opt/pelias/pelias-api/index.js
pelias    1181   959  0 Feb19 ?        00:00:02 /usr/local/nodejs-binary-0.12.2/bin/node --debug-port=5862 /opt/pelias/pelias-api/index.js
pelias    1182   959  0 Feb19 ?        00:00:02 /usr/local/nodejs-binary-0.12.2/bin/node --debug-port=5863 /opt/pelias/pelias-api/index.js
pelias    1183   959  0 Feb19 ?        00:00:02 /usr/local/nodejs-binary-0.12.2/bin/node --debug-port=5864 /opt/pelias/pelias-api/index.js
pelias    1192   959  0 Feb19 ?        00:00:02 /usr/local/nodejs-binary-0.12.2/bin/node --debug-port=5865 /opt/pelias/pelias-api/index.js
pelias    1197   959  0 Feb19 ?        00:00:02 /usr/local/nodejs-binary-0.12.2/bin/node --debug-port=5866 /opt/pelias/pelias-api/index.js
pelias    2293  2144  0 04:43 pts/0    00:00:00 time node index.js
pelias    2294  2293  3 04:43 pts/0    00:00:22 node index.js
pelias    2299  2294 87 04:43 pts/0    00:08:35 /opt/pelias/osm/releases/20160219224954/node_modules/pbf2json/build/pbf2json.linux-x64 -tags=addr:housenumber+addr:street,amenity+name,building+name,shop+name,office+name,public_transport+name,cuisine+name,railway+name,sport+name,natural+name,tourism+name,leisure+name,historic+name,man_made+name,landuse+name,waterway+name,aerialway+name,aeroway+name,craft+name,military+name -leveldb=/opt/pelias/leveldb/osm /opt/pelias/data/osm/planet-latest.osm.pbf

[dianashk]

Sorry you're having so much trouble. Just rereading the whole thread and it seems like you had a successful setup previously until our recent wof integration efforts.

It's not documented anywhere, but we currently use master branches as the development area, so while things on there are working in our dev environments, they aren't blessed for external use and are definitely not production server ready.

I think your best bet is to switch to the production branches for all the pelias components and not risk being caught in the middle of our ongoing development efforts. We will do our best to make sure the docs are updated properly and everything works smoothly once the development of this is completed and it makes its way to the production branch.

[LDuderino-zz]

@dianashk I was using master only because @orangejulius told me to use commit 8c6aba1 on Feb 12, 2016.

Couple of questions for you:

1. How do I get a stable production branch?
2. When I am on production do I need the WOF data and its dependency?

We rely on mapzen/pelias for our geo services at our startup. We are stalled until this gets resolved. Any help is appreciated.

[dianashk]

If you have cloned all the repos to you environment and are running things from there, you'll need to go to each of those repos and execute git checkout production. That should switch the code over to the production branch and you should be able to run the import just like you used to. Production branches currently do not use the new wof data, but still rely on quattroshapes.

If you're using the vagrant setup, it's unfortunately unable to switch to production at the moment and will require a bit of attention from the core team to get it working on production branches.

In the future, once we do merge the updates to production branches, it might be best to test out a build side by side with your production environment, to avoid breaking your workflow in case errors pop up.

Again, sorry about the trouble, as we currently are focusing all of our attention on the hosted mapzen search service and haven't addressed pelias installation and update support.

[LDuderino-zz]

@dianashk Thanks for the update.

Yes, unfortunately we use vagrant setup :(

What is the ETA for your fixes? Just curious if there are other people who rely on this service facing similar issues?

On other note, I understand that your efforts are focused on mapzen search, but once you have some cycles, would it be possible for someone on your team to have a google hangout screenshare session? It might be something simple for someone with expertise and familiarity with the code to figure out why the admin fields are not being populated.

[dianashk]

Hey @LDuderino, sorry for the delayed response. We were able to update the vagrant setup to point to the production branches so grabbing the updates in that repo should get you back to where you were previously.

We would be happy to setup a hangout screenshare session to help you get past the admin lookup problems you're having. We can set something up for later this week.

[LDuderino-zz]

Hi @dianashk, Thanks for the update. How do I get the production snapshot on vagrant? I would like to try that first. Later in the week sounds great for a joint screenshare debug session. Who will be available for that and what times work for you? I can send a google hangout invite accordingly. Currently, we are stalled so whatever works for you, works for me.

Thanks, Neil

[dianashk]

Let's get in touch directly via our pelias@mapzen.com to set this up. Just cloning the master branch of the vagrant repository should get you the update. On Mon, Mar 7, 2016 at 8:03 PM NT notifications@github.com wrote:

Hi @dianashk https://github.com/dianashk, Thanks for the update. How do I get the production snapshot on vagrant? I would like to try that first. Later in the week sounds great for a joint screenshare debug session. Who will be available for that and what times work for you? I can send a google hangout invite accordingly. Currently, we are stalled so whatever works for you, works for me.

Thanks, Neil

— Reply to this email directly or view it on GitHub https://github.com/pelias/whosonfirst/issues/38#issuecomment-193535173.

[LDuderino-zz]

email sent to the aforementioned address.