search

mediagis / nominatim-docker

100% working container for Nominatim
Creative Commons Zero v1.0 Universal
1.07k stars 437 forks source link

Different results with nominatim website and the local instance. #446

Closed kvermeul closed 10 months ago

kvermeul commented 1 year ago

Describe the bug Hey folks, not sure it is the right place to post this behavior, so feel free to redirect me to where I should post this message if needed.

The problem is that I am having different results for the same nominatim query ran on the nominatim website and in local.

To Reproduce Steps to reproduce the behavior:

I first downloaded the midi-pyrenees-latest.osm.pbf and the wikimedia-importance.sql.gz.

docker run -it \ -v /storage/nominatim-data:/nominatim/data \ -e PBF_PATH=/nominatim/data/midi-pyrenees-latest.osm.pbf \ -e IMPORT_WIKIPEDIA=/nominatim/data/wikimedia-importance.sql.gz \ -e IMPORT_STYLE=extratags \ -e THREADS=64 \ -e shm-size=8192M \ -e USER_AGENT="some_user_agent" // to fix curl 403 error -p 8080:8080 \ --name nominatim \ mediagis/nominatim:4.2

When I run this request:

https://nominatim.openstreetmap.org/search?q=university&format=json&bounded=1&viewbox=1.3581,43.5275,1.5024,43.6344&limit=10

I get the following results: [{"place_id":120596823,"licence":"Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright","osm_type":"way","osm_id":59941415,"boundingbox":["43.5558473","43.5722494","1.4616498","1.4766743"],"lat":"43.564215000000004","lon":"1.4665760358580304","display_name":"Université Paul Sabatier - Toulouse III, Route de Narbonne, Rangueil, Sauzelong, Pech David, Pouvourville, Toulouse Sud-Est, Toulouse, Haute-Garonne, Occitanie, France métropolitaine, 31400, France","class":"amenity","type":"university","importance":0.43834980554030356,"icon":"https://nominatim.openstreetmap.org/ui/mapicons/education_university.p.20.png"},{"place_id":136861512,"licence":"Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright","osm_type":"way","osm_id":133729864,"boundingbox":["43.577829","43.58078","1.462077","1.4654204"],"lat":"43.5792661","lon":"1.4637364030461741","display_name":"CEMES - CNRS, Avenue de Lespinet, Pont de Demoiselles, Toulouse Sud-Est, Toulouse, Haute-Garonne, Occitanie, France métropolitaine, 31400, France","class":"amenity","type":"university","importance":0.042720323856182026,"icon":"https://nominatim.openstreetmap.org/ui/mapicons/education_university.p.20.png"},{"place_id":123313654,"licence":"Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright","osm_type":"way","osm_id":74688260,"boundingbox":["43.5774985","43.57793","1.4053676","1.405694"],"lat":"43.57771375","lon":"1.4055305057540317","display_name":"Henry Mayer, Allée Antonio Machado, Mirail - Reynerie - Bellefontaine, Toulouse Ouest, Toulouse, Haute-Garonne, Occitanie, France métropolitaine, 31100, France","class":"amenity","type":"university","importance":0.0001,"icon":"https://nominatim.openstreetmap.org/ui/mapicons/education_university.p.20.png"},{"place_id":121407368,"licence":"Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright","osm_type":"way","osm_id":64527121,"boundingbox":["43.5987547","43.5989253","1.4408807","1.4413525"],"lat":"43.598845350000005","lon":"1.4411174807499987","display_name":"MJM Graphic Design, Descente de la Halle Aux Poissons, Place Occitane, Capitole, Toulouse Centre, Toulouse, Haute-Garonne, Occitanie, France métropolitaine, 31000, France","class":"amenity","type":"university","importance":0.0001,"icon":"https://nominatim.openstreetmap.org/ui/mapicons/education_university.p.20.png"},{"place_id":298458373,"licence":"Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright","osm_type":"relation","osm_id":6158485,"boundingbox":["43.5774024","43.5787104","1.3998158","1.4004848"],"lat":"43.578079349999996","lon":"1.4001520455014749","display_name":"Bâtiment Philippe Malrieu, Est/Ouest 5a, Mirail - Reynerie - Bellefontaine, Toulouse Ouest, Toulouse, Haute-Garonne, Occitanie, France métropolitaine, 31100, France","class":"amenity","type":"university","importance":0.0001,"icon":"https://nominatim.openstreetmap.org/ui/mapicons/education_university.p.20.png"},{"place_id":298426091,"licence":"Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright","osm_type":"relation","osm_id":6158484,"boundingbox":["43.5781549","43.5795484","1.4009872","1.4019123"],"lat":"43.578918","lon":"1.4013865941217687","display_name":"Bâtiment Olympe de Gouges, Nord/Sud 3, Mirail - Reynerie - Bellefontaine, Toulouse Ouest, Toulouse, Haute-Garonne, Occitanie, France métropolitaine, 31100, France","class":"amenity","type":"university","importance":0.0001,"icon":"https://nominatim.openstreetmap.org/ui/mapicons/education_university.p.20.png"},{"place_id":298009617,"licence":"Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright","osm_type":"relation","osm_id":1799670,"boundingbox":["43.593593","43.594621","1.44949","1.45134"],"lat":"43.5941045","lon":"1.4510750559100802","display_name":"Université Toulouse III Faculté de médecine Toulouse Purpan, Allée Matilda, Saint-Michel, Le Busca, Empalot, Saint-Agne, Toulouse Sud-Est, Toulouse, Haute-Garonne, Occitanie, France métropolitaine, 31400, France","class":"amenity","type":"university","importance":0.0001,"icon":"https://nominatim.openstreetmap.org/ui/mapicons/education_university.p.20.png"},{"place_id":255654127,"licence":"Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright","osm_type":"way","osm_id":762822133,"boundingbox":["43.5616265","43.5618464","1.469653","1.4700078"],"lat":"43.561736499999995","lon":"1.4698510652926413","display_name":"UPSSITECH, 118, Route de Narbonne, Rangueil, Sauzelong, Pech David, Pouvourville, Toulouse Sud-Est, Toulouse, Haute-Garonne, Occitanie, France métropolitaine, 31062, France","class":"amenity","type":"university","importance":9.99999999995449e-6,"icon":"https://nominatim.openstreetmap.org/ui/mapicons/education_university.p.20.png"},{"place_id":270774171,"licence":"Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright","osm_type":"way","osm_id":855562283,"boundingbox":["43.579212","43.5793603","1.4021606","1.4024109"],"lat":"43.5792722","lon":"1.4022856537847632","display_name":"Amphi MDS1, Est/Ouest 5c, Mirail - Reynerie - Bellefontaine, Toulouse Ouest, Toulouse, Haute-Garonne, Occitanie, France métropolitaine, 31058, France","class":"amenity","type":"university","importance":9.99999999995449e-6,"icon":"https://nominatim.openstreetmap.org/ui/mapicons/education_university.p.20.png"},{"place_id":356425469,"licence":"Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright","osm_type":"node","osm_id":10657896491,"boundingbox":["43.5767596","43.5768596","1.4006282","1.4007282"],"lat":"43.5768096","lon":"1.4006782","display_name":"Service commun universitaire d'information, d'orientation et d'insertion professionnelle, 5, Allée Antonio Machado, Mirail - Reynerie - Bellefontaine, Toulouse Ouest, Toulouse, Haute-Garonne, Occitanie, France métropolitaine, 31058, France","class":"amenity","type":"university","importance":9.99999999995449e-6,"icon":"https://nominatim.openstreetmap.org/ui/mapicons/education_university.p.20.png"}]

So I would expect to have the same behavior with my local instance.

However, what I observe when I do curl "http://localhost:8080/search?q=university&format=json&bounded=1&viewbox=1.3581,43.5275,1.5024,43.6344&limit=10"

is the following answer:

[{"place_id":164339,"licence":"Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright","osm_type":"node","osm_id":2460954143,"boundingbox":["43.600169","43.600269","1.4408079","1.4409079"],"lat":"43.600219","lon":"1.4408579","display_name":"Stephan university, Rue Peyrolières, Place Occitane, Capitole, Toulouse Centre, Toulouse, Haute-Garonne, 31000, France","class":"shop","type":"hairdresser","importance":0.10000999999999996}]

To further investigate, I double checked that the osm_id returned by the nominatim website are present in my midi-pyrenees-latest.osm.pbf file with osmium and they actually are in it.

Also, I do not know if this is related, but I observed tgat the nominatim website returns places with osm_type=way, whereas the local instance returned a place with an osm_type=node.

Here are the logs of the server if it can help

nominatim.logs.txt

Desktop / Server (please complete the following information):

Additional context Add any other context about the problem here.

mtmail commented 1 year ago

nominatim.osm.org runs unreleased code changes (git master branch) but also an older database. It's not possible to reproduce results exactly. But I think with "university" and viewbox it's a category phrase in this case.

Try running the command in the "Adding search through category phrases" section of https://nominatim.org/release-docs/latest/admin/Import/

kvermeul commented 1 year ago

Thanks for the pointer!

So what I did is that I ran the command inside the docker container (after fixing some postgres permissions, got it running): nominatim special-phrases --import-from-wiki

The logs of the command:

2023-04-22 13:35:33: Using project directory: /app 2023-04-22 13:35:36: Special phrases importation starting 2023-04-22 13:35:36: Importing phrases for lang: af... 2023-04-22 13:35:36: Importing phrases for lang: ar... 2023-04-22 13:35:37: Importing phrases for lang: br... 2023-04-22 13:35:37: Importing phrases for lang: ca... 2023-04-22 13:35:37: Importing phrases for lang: cs... 2023-04-22 13:35:37: Importing phrases for lang: de... 2023-04-22 13:35:38: Importing phrases for lang: en... 2023-04-22 13:35:38: Importing phrases for lang: es... 2023-04-22 13:35:38: Importing phrases for lang: et... 2023-04-22 13:35:38: Importing phrases for lang: eu... 2023-04-22 13:35:39: Importing phrases for lang: fa... 2023-04-22 13:35:39: Importing phrases for lang: fi... 2023-04-22 13:35:39: Importing phrases for lang: fr... 2023-04-22 13:35:39: Importing phrases for lang: gl... 2023-04-22 13:35:39: Importing phrases for lang: hr... 2023-04-22 13:35:40: Importing phrases for lang: hu... 2023-04-22 13:35:40: Importing phrases for lang: ia... 2023-04-22 13:35:40: Importing phrases for lang: is... 2023-04-22 13:35:40: Importing phrases for lang: it... 2023-04-22 13:35:41: Importing phrases for lang: ja... 2023-04-22 13:35:41: Importing phrases for lang: mk... 2023-04-22 13:35:41: Importing phrases for lang: nl... 2023-04-22 13:35:41: Importing phrases for lang: no... 2023-04-22 13:35:41: Importing phrases for lang: pl... 2023-04-22 13:35:42: Bad class/type: shop=. It will not be imported 2023-04-22 13:35:42: Bad class/type: shop=. It will not be imported 2023-04-22 13:35:42: Bad class/type: shop=. It will not be imported 2023-04-22 13:35:42: Bad class/type: shop=. It will not be imported 2023-04-22 13:35:42: Bad class/type: shop=. It will not be imported 2023-04-22 13:35:42: Bad class/type: shop=. It will not be imported 2023-04-22 13:35:42: Importing phrases for lang: ps... 2023-04-22 13:35:42: Importing phrases for lang: pt... 2023-04-22 13:35:42: Importing phrases for lang: ru... 2023-04-22 13:35:43: Importing phrases for lang: sk... 2023-04-22 13:35:43: Importing phrases for lang: sl... 2023-04-22 13:35:43: Importing phrases for lang: sv... 2023-04-22 13:35:43: Importing phrases for lang: uk... 2023-04-22 13:35:44: Importing phrases for lang: vi... 2023-04-22 13:35:44: Create tables and indexes... 2023-04-22 13:35:51: Cleaning database... 2023-04-22 13:35:55: Import done. 2023-04-22 13:35:55: 6 phrases were invalid and have been skipped during the whole process.

Unfortunately, the result of the query after this (with and without a restart of the container) is the same: no results in common with the website.

Actually, I just need to know how to get the same results with the local instance (i.e., how to retrieve the universities in the viewbox), the query needs not to be the same. Maybe I am not using the right query?

ms-webdev commented 1 year ago

Hi, i have the same problem. With this hack inner the docker container terminal:

su - nominatim
nominatim special-phrases --import-from-wiki

the import is running but the search api ignores the imported special-phrases. @kvermeul do you have solve the problem?

EDIT I solved it, not nice but it works. Single words from the wiki-spacial-phrases doesnt have an operator (in, near ...). After the wiki-phrases import (nominatim special-phrases --import-from-wiki) I updated the nominatim table word and replace the NULL operators with "in":

su - nominatim
psql -d nominatim
UPDATE word SET info = jsonb_set(info, '{op}', '"in"') WHERE info->>'op' IS NULL and type = 'S';
mtmail commented 1 year ago

nominatim.openstreetmap.org currently run a different, unreleased, version and can't be direct compared to the docker installation (Nominatim 4.2.3). https://nominatim.org/2023/08/13/going-live.html

@ms-webdev If you have a suggestion to improve the documentation then https://github.com/osm-search/nominatim is the correct repository (https://github.com/osm-search/Nominatim/edit/master/docs/customize/Special-Phrases.md).