search

pelias / pip-service

Pelias point-in-polygon-service
https://pelias.io
MIT License
16 stars 18 forks source link

Locality not returned #101

Closed TariqAHassan closed 5 years ago

TariqAHassan commented 5 years ago

Hi @orangejulius.

First, sorry to ping you directly, but I have been unable to resolve this issue for a while and am really quite confused about what is going on. In short, I have been unable to get the current version, or even prior versions, of this package to reliably return the locaility.

While it is true for the example on the homepage:

curl http://localhost:3102/-106.937/34.060

The following query for Austin TX only returns the neighbourhood:

curl http://localhost:3102/-97.7431/30.2672

Indeed, I found this to be true in many places in North America (try anywhere in Toronto, for example), at least w.r.t. my limited testing.

Of course, it is possible to solve for the locality by looking up the, say, neighbourhood's hierarchy in the WoF data...however, this is very tedious to implement.

Any idea on why this is happening? Is this happening by design?

Edit:

Joxit commented 5 years ago

Hello,

Is data/101/724/577/101724577.geojson present in your wof bundle path ?

TariqAHassan commented 5 years ago

Hi. I do not have a directory data/101/724.

In case it is of any help, I did an admin only download.

Joxit commented 5 years ago

That's means you used --admin-only without imports.whosonfirst.importPlace right ?

That meas your missing locality may be related to https://github.com/whosonfirst/go-whosonfirst-dist/issues/9

orangejulius commented 5 years ago

Hey @TariqAHassan, Thanks for the report. As @Joxit has confirmed, this does look like an issue with the WOF bundles. I spun up a PIP service instance with fresh WOF data and am getting the same thing, and can confirm the Austin, TX record is missing from the WOF bundle completely.

I think a fix is coming fairly soon from the WOF team, but @joxit is also working on https://github.com/pelias/whosonfirst/pull/417 which looks like a promising solution.

Worst case, it is possible to download WOF data directly from GitHub using git-lfs. It's pretty disk space and inode intensive, but depends on the least amount of processing. We really have tried to steer people away from using git-lfs for a while, but I wonder if it's worth revisiting that.

TariqAHassan commented 5 years ago

@Joxit . Yes, that is correct.

Thanks to both of you for the help. I am planning to rebuild against the wof data repo directly and report back.

TariqAHassan commented 5 years ago

Hi.

I just finished building pelias (the whole geocoder) against a direct clone from the whosonfirst-data/whosonfirst-data repo.

The locality issues have gone away. Thanks again to both of you for the guidances.

bboure commented 4 years ago

Hi,

I am experiencing the same issue. After checking the compare tool, it does not seem like an issue in my build. Locality is also missing in https://api.geocode.earth

Places in Brussels do not get the locality property.

missinglink commented 4 years ago

@bboure your issue seems to be due to neighbourhood:85801105 being parented by locality:101751573 which is 'both deprecated and superseded by: 1175610569'.

Can you please comment on https://github.com/whosonfirst-data/whosonfirst-data/issues/1377 to let the WOF team know?

bboure commented 4 years ago

@orangejulius ok thanks

missinglink commented 4 years ago

Hi @bboure I've written a script to fix the outdated WOF hierarchies. I'd appreciate if you could test out my fix before I write the code to patch all the files on the CDN?

curl -O https://data.geocode.earth/wof/dist/sqlite/whosonfirst-data-admin-be-latest.db.bz2

lbunzip2 whosonfirst-data-admin-be-latest.db.bz2
npm install --global @whosonfirst/wof@latest
+ @whosonfirst/wof@0.26.0
wof sqlite fix whosonfirst-data-admin-be-latest.db
....
85801105 has an incorrect parent_id, replacing 101751573 with 1175610569
85801105 has an incorrect county_id, replacing 102049957 with 85681713
85801105 has an incorrect locality_id, replacing 101751573 with 1175610569
85801105 has an incorrect macroregion_id, replacing 404227355 with 85681713
....
bboure commented 4 years ago

Thank you @missinglink I will have a look at this

bboure commented 4 years ago

@orangejulius after executing the script and re-importing OSM records, I can confirm the locality is now showing correctly.

        "country": "Belgium",
        "country_gid": "whosonfirst:country:85632997",
        "country_a": "BEL",
        "region": "Brussels-Capital Region",
        "region_gid": "whosonfirst:region:85681713",
        "region_a": "BU",
        "locality": "Brussel",
        "locality_gid": "whosonfirst:locality:1175610569",

On a side note, the wof-admin-lookup service is getting lots of duplicates in Brussels. Not sure if this is a related issue.

Thanks!!