[question] Query for "is provided point inside populated area?"

[kradem]

Hi everybody!

I was using Pelias in development for some time, but I haven't figured out what to search if I want some kind of easy reverse geocoding response: is this point (lon-lat) inside populated area or not?

What would be the best practice for that and by which source? Am I confident if I, for example, condition my query to wof source, try to get locality attribute and check for the existence of wk:population entry?

Btw, this response I need is just for informational purposes and correct answers aren't crucial, I'd rather be implementing some fast queries with less accuracy than slow sophisticated algorithms.

Thank you very much for your work!

[orangejulius]

Hi @kradem, The /v1/reverse endpoint is definitely the one you want. However its ability to do coarse (admin area only, as opposed to addresses and venues too) reverse geocoding has been pretty poor for a long time.

Fortunately, we are currently rolling out a new point in polygon service that does an excellent job at coarse reverse. You could try setting that up on its own, or use it with the pelias api (support was recently merged to the master branch).

This new service should be able to always return, for any given lat/lon, what whosonfirst admin areas that point is within. Using the population from the corresponding full record like you suggested would be a great way to determine if a place is populated.

Please let us know how it goes and don't hesitate to ask if you have any more questions!

[kradem]

Regards!

The /v1/reverse endpoint is definitely the one you want. However its ability to do coarse (admin area only, as opposed to addresses and venues too) reverse geocoding has been pretty poor for a long time.

This explains a lot of things happened last few months in my project's tests... :-)

Fortunately, we are currently rolling out a new point in polygon service that does an excellent job at coarse reverse. You could try setting that up on its own, or use it with the pelias api (support was recently merged to the master branch).

This new service should be able to always return, for any given lat/lon, what whosonfirst admin areas that point is within. Using the population from the corresponding full record like you suggested would be a great way to determine if a place is populated.

Something went wrong with the last libpostal upgrade I had on my development machine, but I would try remove it as textAnalyzer and look for this brand new feature.

Please let us know how it goes and don't hesitate to ask if you have any more questions!

As soon I have anything to report I'll write here.

Thank you very much!

[kradem]

Hey @orangejulius!

I'm stuck with this. I successfully installed api@master and if I understand correctly api's code (btw, prior to installing and using Pelias I have never been dealing with node.js...) pip service has to be enabled (I can't find any configuration entry for this) and reverse layers attribute has to be coarse, then pip-service would automatically run.

For example, if I send this request to running Pelias API service:

curl 'http://127.0.0.1:3100/v1/reverse?point.lat=30.187&point.lon=71.499&layers=coarse&sources=wof'

that should be equal to request on running standalone pip-service:

curl 'http://127.0.0.1:3102/71.499/30.187'

I'm afraid I can't say that happens on my development machine and I cannot manage to run standalone service:

Successful install

``` ~/repos/pelias/pip-service$ npm install npm WARN deprecated npmconf@2.1.2: this package has been reintegrated into npm and is now out of date with respect to npm npm WARN deprecated @bahmutov/parse-github-repo-url@0.1.2: All features were merged back into original parse-github-repo-url > git-validate@2.2.2 install /home/kradem/repos/pelias/pip-service/node_modules/precommit-hook/node_modules/git-validate > node bin/install > precommit-hook@3.0.0 install /home/kradem/repos/pelias/pip-service/node_modules/precommit-hook > node bin/install npm WARN deprecated conventional-changelog@0.0.17: Please update conventional-changelog to >1.0.0. If you are running the cli, use conventional-changelog-cli npm WARN deprecated node-uuid@1.4.8: Use uuid module instead lodash@4.17.4 node_modules/lodash temp@0.8.3 node_modules/temp ├── os-tmpdir@1.0.2 └── rimraf@2.2.8 precommit-hook@3.0.0 node_modules/precommit-hook └── git-validate@2.2.2 tap-dot@1.0.5 node_modules/tap-dot ├── chalk@1.1.3 (escape-string-regexp@1.0.5, supports-color@2.0.0, ansi-styles@2.2.1, strip-ansi@3.0.1, has-ansi@2.0.0) ├── through2@2.0.3 (xtend@4.0.1, readable-stream@2.2.9) └── tap-out@1.4.2 (trim@0.0.1, re-emitter@1.1.3, split@1.0.0, readable-stream@2.2.9) proxyquire@1.7.11 node_modules/proxyquire ├── module-not-found-error@1.0.1 ├── resolve@1.1.7 └── fill-keys@1.0.2 (merge-descriptors@1.0.1, is-object@1.0.1) tape@4.6.3 node_modules/tape ├── inherits@2.0.3 ├── resumer@0.0.0 ├── has@1.0.1 ├── defined@1.0.0 ├── deep-equal@1.0.1 ├── function-bind@1.1.0 ├── through@2.3.8 ├── minimist@1.2.0 ├── object-inspect@1.2.2 ├── resolve@1.1.7 ├── glob@7.1.1 (path-is-absolute@1.0.1, fs.realpath@1.0.0, once@1.4.0, inflight@1.0.6, minimatch@3.0.3) ├── for-each@0.3.2 (is-function@1.0.1) └── string.prototype.trim@1.1.2 (define-properties@1.1.2, es-abstract@1.7.0) pelias-wof-admin-lookup@3.5.0 node_modules/pelias-wof-admin-lookup ├── simplify-js@1.2.1 ├── pelias-parallel-stream@0.1.0 ├── csv-parse@1.2.0 ├── through2-filter@2.0.0 (xtend@4.0.1) ├── through2-map@3.0.0 (xtend@4.0.1) ├── through2@2.0.3 (xtend@4.0.1, readable-stream@2.2.9) ├── pelias-config@2.9.0 (mergeable@0.0.0) ├── async@2.3.0 ├── joi@10.4.1 (topo@2.0.2, items@2.1.1, isemail@2.2.1, hoek@4.1.1) ├── polygon-lookup@2.2.0 (point-in-polygon@1.0.1, rbush@2.0.1) ├── through2-sink@1.0.0 (xtend@3.0.0, through2@0.5.1) └── pelias-logger@0.2.0 (winston@2.3.1) express@4.15.2 node_modules/express ├── setprototypeof@1.0.3 ├── escape-html@1.0.3 ├── array-flatten@1.1.1 ├── utils-merge@1.0.0 ├── cookie-signature@1.0.6 ├── merge-descriptors@1.0.1 ├── content-type@1.0.2 ├── methods@1.1.2 ├── encodeurl@1.0.1 ├── serve-static@1.12.1 ├── fresh@0.5.0 ├── etag@1.8.0 ├── content-disposition@0.5.2 ├── vary@1.1.1 ├── range-parser@1.2.0 ├── parseurl@1.3.1 ├── path-to-regexp@0.1.7 ├── cookie@0.3.1 ├── statuses@1.3.1 ├── depd@1.1.0 ├── qs@6.4.0 ├── debug@2.6.1 (ms@0.7.2) ├── send@0.15.1 (destroy@1.0.4, ms@0.7.2, mime@1.3.4, http-errors@1.6.1) ├── on-finished@2.3.0 (ee-first@1.1.1) ├── proxy-addr@1.1.4 (forwarded@0.1.0, ipaddr.js@1.3.0) ├── accepts@1.3.3 (negotiator@0.6.1, mime-types@2.1.15) ├── finalhandler@1.0.1 (unpipe@1.0.0, debug@2.6.3) └── type-is@1.6.15 (media-typer@0.3.0, mime-types@2.1.15) jshint@2.9.4 node_modules/jshint ├── strip-json-comments@1.0.4 ├── exit@0.1.2 ├── minimatch@3.0.3 (brace-expansion@1.1.7) ├── shelljs@0.3.0 ├── console-browserify@1.1.0 (date-now@0.1.4) ├── cli@1.0.1 (glob@7.1.1) ├── htmlparser2@3.8.3 (domelementtype@1.3.0, entities@1.0.0, domhandler@2.3.0, readable-stream@1.1.14, domutils@1.5.1) └── lodash@3.7.0 npm-check@5.4.0 node_modules/npm-check ├── semver-diff@2.1.0 ├── co@4.6.0 ├── semver@5.3.0 ├── giturl@1.0.0 ├── throat@2.0.2 ├── text-table@0.2.0 ├── chalk@1.1.3 (escape-string-regexp@1.0.5, ansi-styles@2.2.1, supports-color@2.0.0, has-ansi@2.0.0, strip-ansi@3.0.1) ├── node-emoji@1.5.1 (string.prototype.codepointat@0.2.0) ├── is-ci@1.0.10 (ci-info@1.0.0) ├── path-exists@2.1.0 (pinkie-promise@2.0.1) ├── pkg-dir@1.0.0 (find-up@1.1.2) ├── minimatch@3.0.3 (brace-expansion@1.1.7) ├── merge-options@0.0.64 (is-plain-obj@1.1.0) ├── globby@4.1.0 (arrify@1.0.1, object-assign@4.1.1, pify@2.3.0, pinkie-promise@2.0.1, glob@6.0.4, array-union@1.0.2) ├── global-modules@0.2.3 (is-windows@0.2.0, global-prefix@0.1.5) ├── update-notifier@0.6.3 (is-npm@1.0.0, latest-version@2.0.0, boxen@0.3.1, configstore@2.1.0) ├── ora@0.2.3 (object-assign@4.1.1, cli-spinners@0.1.2, cli-cursor@1.0.2) ├── execa@0.2.2 (strip-eof@1.0.0, path-key@1.0.0, npm-run-path@1.0.0, object-assign@4.1.1, cross-spawn-async@2.2.5) ├── inquirer@0.12.0 (strip-ansi@3.0.1, ansi-regex@2.1.1, ansi-escapes@1.4.0, through@2.3.8, rx-lite@3.1.2, cli-width@2.1.0, figures@1.7.0, cli-cursor@1.0.2, string-width@1.0.2, run-async@0.1.0, readline2@1.0.1) ├── package-json@2.4.0 (registry-url@3.1.0, registry-auth-token@3.1.2, got@5.7.1) ├── callsite-record@3.2.2 (callsite@1.0.0, pinkie-promise@2.0.1, highlight-es@1.0.1, error-stack-parser@1.3.6) ├── meow@3.7.0 (decamelize@1.2.0, map-obj@1.0.1, trim-newlines@1.0.0, object-assign@4.1.1, minimist@1.2.0, camelcase-keys@2.1.0, loud-rejection@1.6.0, redent@1.0.0, normalize-package-data@2.3.6, read-pkg-up@1.0.1) ├── depcheck@0.6.7 (deps-regex@0.1.4, require-package-name@2.0.1, builtin-modules@1.1.1, deprecate@1.0.0, babylon@6.16.1, walkdir@0.0.11, babel-traverse@6.24.1, js-yaml@3.8.3, yargs@6.6.0) └── babel-runtime@6.23.0 (regenerator-runtime@0.10.3, core-js@2.4.1) semantic-release@6.3.2 node_modules/semantic-release ├── require-relative@0.8.7 ├── @semantic-release/error@1.0.0 ├── @bahmutov/parse-github-repo-url@0.1.2 ├── semver@5.3.0 ├── nerf-dart@1.0.0 ├── run-series@1.1.4 ├── nopt@3.0.6 (abbrev@1.1.0) ├── github@0.2.4 (mime@1.3.4) ├── run-auto@2.0.0 (dezalgo@1.0.3) ├── npmconf@2.1.2 (inherits@2.0.3, ini@1.3.4, uid-number@0.0.5, once@1.3.3, semver@4.3.6, config-chain@1.1.11, osenv@0.1.4, mkdirp@0.5.1) ├── npmlog@4.0.2 (console-control-strings@1.1.0, set-blocking@2.0.0, gauge@2.7.3, are-we-there-yet@1.1.2) ├── normalize-package-data@2.3.6 (hosted-git-info@2.4.2, validate-npm-package-license@3.0.1, is-builtin-module@1.0.0) ├── git-head@1.20.1 (git-refs@1.1.3) ├── @semantic-release/last-release-npm@1.2.1 (npmlog@1.2.1, npm-registry-client@7.5.0) ├── @semantic-release/commit-analyzer@2.0.0 (conventional-changelog@0.0.17) ├── @semantic-release/release-notes-generator@2.0.0 (github-url-from-git@1.5.0, conventional-changelog@0.0.17) └── @semantic-release/condition-travis@5.0.2 (travis-deploy-once@1.0.0-node-0.10-support) request@2.81.0 node_modules/request ├── aws-sign2@0.6.0 ├── oauth-sign@0.8.2 ├── forever-agent@0.6.1 ├── tunnel-agent@0.6.0 ├── caseless@0.12.0 ├── is-typedarray@1.0.0 ├── stringstream@0.0.5 ├── safe-buffer@5.0.1 ├── aws4@1.6.0 ├── isstream@0.1.2 ├── json-stringify-safe@5.0.1 ├── extend@3.0.0 ├── performance-now@0.2.0 ├── uuid@3.0.1 ├── qs@6.4.0 ├── mime-types@2.1.15 (mime-db@1.27.0) ├── combined-stream@1.0.5 (delayed-stream@1.0.0) ├── form-data@2.1.4 (asynckit@0.4.0) ├── tough-cookie@2.3.2 (punycode@1.4.1) ├── hawk@3.1.3 (cryptiles@2.0.5, boom@2.10.1, sntp@1.0.9, hoek@2.16.3) ├── http-signature@1.1.1 (assert-plus@0.2.0, jsprim@1.4.0, sshpk@1.13.0) └── har-validator@4.2.1 (har-schema@1.0.5, ajv@4.11.6) ```

Fail to run

``` ~/repos/pelias/pip-service$ npm start > pelias-pip-service@0.0.0-development start /home/kradem/repos/pelias/pip-service > node index.js [TypeError: Path must be a string. Received undefined] npm ERR! Linux 4.4.0-72-generic npm ERR! argv "/usr/bin/nodejs" "/usr/bin/npm" "start" npm ERR! node v4.8.2 npm ERR! npm v2.15.11 npm ERR! code ELIFECYCLE npm ERR! pelias-pip-service@0.0.0-development start: `node index.js` npm ERR! Exit status 1 npm ERR! npm ERR! Failed at the pelias-pip-service@0.0.0-development start script 'node index.js'. npm ERR! This is most likely a problem with the pelias-pip-service package, npm ERR! not with npm itself. npm ERR! Tell the author that this fails on your system: npm ERR! node index.js npm ERR! You can get information on how to open an issue for this project with: npm ERR! npm bugs pelias-pip-service npm ERR! Or if that isn't available, you can get their info via: npm ERR! npm ERR! npm owner ls pelias-pip-service npm ERR! There is likely additional logging output above. npm ERR! Please include the following file with any support request: npm ERR! /home/kradem/repos/pelias/pip-service/npm-debug.log ```

[orangejulius]

@kradem I think the pip service needs the path to WOF data as a parameter. so something like npm start -- /path/to/wof/data. @trescube can we change that to be read from pelias.json? (if there was a good reason for it I'm forgetting let me know)

[trescube]

The syntax is npm start <path to WOF data>. The project doesn't currently use pelias-config since it's a heavy to come along for the ride for a single value. Support for a command line parameter and config could probably exist, though.

[kradem]

Sorry for delay!

Yep, of course, I haven't provided wof source. So, I have got running pip-service ready to give me some point information:

This is a request for a point at sea:

curl 'http://127.0.0.1:3102/14.94937115/43.892623'

{
    "region": [{
        "id": 85684785,
        "name": "Inland waters of Croatia",
        "centroid": {
            "lat": 43.618249,
            "lon": 15.646464
        },
        "bounding_box": "13.224042,42.1758995,18.5415705,45.5030665"
    }],
    "country": [{
        "id": 85633229,
        "name": "Croatia",
        "abbr": "HRV",
        "centroid": {
            "lat": 45.067726,
            "lon": 16.398549
        },
        "bounding_box": "13.501475,42.416327,19.407838,46.546979"
    }]
}

and this is for some mountain point:

curl 'http://127.0.0.1:3102/15.072006/44.577639'

{
    "region": [{
        "id": 85684737,
        "name": "Licko-senjska",
        "centroid": {
            "lat": 44.687678,
            "lon": 15.410409
        },
        "bounding_box": "14.733,44.271229,16.13929,45.1228905"
    }],
    "country": [{
        "id": 85633229,
        "name": "Croatia",
        "abbr": "HRV",
        "centroid": {
            "lat": 45.067726,
            "lon": 16.398549
        },
        "bounding_box": "13.501475,42.416327,19.407838,46.546979"
    }]
}

both with the lack of locality attribute.

The next examples have locality attributes:

curl 'http://127.0.0.1:3102/14.901770/44.988023'

{
    "locality": [{
        "id": 101795793,
        "name": "Senj",
        "centroid": {
            "lat": 44.990145,
            "lon": 14.904972
        },
        "bounding_box": "14.8950382746,44.9802379292,14.9215760792,45.0072016039"
    }],
    "region": [{
        "id": 85684737,
        "name": "Licko-senjska",
        "centroid": {
            "lat": 44.687678,
            "lon": 15.410409
        },
        "bounding_box": "14.733,44.271229,16.13929,45.1228905"
    }],
    "country": [{
        "id": 85633229,
        "name": "Croatia",
        "abbr": "HRV",
        "centroid": {
            "lat": 45.067726,
            "lon": 16.398549
        },
        "bounding_box": "13.501475,42.416327,19.407838,46.546979"
    }]
}

curl 'http://127.0.0.1:3102/26.713394/58.372835'

{
    "locality": [{
        "id": 101748151,
        "name": "Tartu",
        "centroid": {
            "lat": 58.373635,
            "lon": 26.731958
        },
        "bounding_box": "26.6709107752,58.3292883301,26.8197040973,58.4080162143"
    }],
    "region": [{
        "id": 85682995,
        "name": "Tartumaa",
        "centroid": {
            "lat": 58.367777,
            "lon": 26.738988
        },
        "bounding_box": "26.0771095,58.113041,27.480841,58.682676"
    }],
    "country": [{
        "id": 85633135,
        "name": "Estonia",
        "abbr": "EST",
        "centroid": {
            "lat": 58.671014,
            "lon": 25.535793
        },
        "bounding_box": "21.832367,57.515819,28.186475,59.670885"
    }]
}

so I suppose I'll just look for the existence of that locality attribute and set the is_populated property according to that.

Also, as I need the point's label too, I'll go with name attribute of locality for populated point, respectively region (if attr exists) or country for the non-populated area.

[orangejulius]

Hello!

Sorry we never got back to you on this, but it looks like you figured everything out. In the past year, the quality of PIP service results should only have improved, and if you seem to be on the right track with checking for a locality record (you might want to consider localadmin as well).

If you have a chance, let us know if it worked out.