search

pelias / docker

Run the Pelias geocoder in docker containers, including example projects.
MIT License
314 stars 217 forks source link

`label` only consists of place `name` in the responses of `search`, `autocomplete` endpoints. #307

Closed theoway closed 1 year ago

theoway commented 1 year ago

Describe the bug Responses of search & autocomplete send incomplete labels that are same as name, that is, only consist of place name and not the address(region, state, country, pincode etc) suffixed to them. The endpoints also throw warning: "warnings":["performance optimization: excluding 'address' layer"] Even after disabling this optimization in pelias.json, this warning still comes.

I am only using OSM data, imported as an osm.pbf file. It is a merged dataset made from merging two different osm.pbf files. My file is not corrupt as I have tested the same file with Nominatim, and it returned completed addresses with each location.

Is this happening because I have not imported WOF data of the same region?

Steps to Reproduce Steps to reproduce the behavior:

  1. Clone this: https://github.com/pelias/docker/tree/master/projects/portland-metro
  2. Ran the commands in the order:
pelias compose pull
pelias elastic start
pelias elastic wait
pelias elastic create
pelias download all
pelias prepare all
pelias import all
pelias compose up
pelias test run
  1. See the results. Note the label and name for each location is same: 
    {
"geocoding":{
  "version":"0.2",
  "attribution":"http://1x9.5x.xx.1xx/attribution",
  "query":{
     "text":"mosq",
     "parser":"pelias",
     "parsed_text":{
        "subject":"mosq"
     },
     "size":20,
     "layers":[
        "venue",
        "street"
     ],
     "private":false,
     "focus.point.lat":45.52,
     "focus.point.lon":-122.67,
     "boundary.rect.min_lat":28.22,
     "boundary.rect.max_lat":29.01,
     "boundary.rect.min_lon":76.69,
     "boundary.rect.max_lon":77.73,
     "lang":{
        "name":"English",
        "iso6391":"en",
        "iso6393":"eng",
        "via":"header",
        "defaulted":false
     },
     "querySize":40
  },
  "warnings":[
     "performance optimization: excluding 'address' layer"
  ],
  "engine":{
     "name":"Pelias",
     "author":"Mapzen",
     "version":"1.0"
  },
  "timestamp":1674800525243
},
"type":"FeatureCollection",
"features":[
  {
     "type":"Feature",
     "geometry":{
        "type":"Point",
        "coordinates":[
           77.220352,
           28.593019
        ]
     },
     "properties":{
        "id":"way/220693076",
        "gid":"openstreetmap:venue:way/220693076",
        "layer":"venue",
        "source":"openstreetmap",
        "source_id":"way/220693076",
        "name":"Bara Gumbad tomb and mosque",
        "distance":11545.161,
        "accuracy":"point",
        "label":"Bara Gumbad tomb and mosque",
        "addendum":{
           "osm":{
              "wikidata":"Q11914956",
              "wikipedia":"en:Bara Gumbad",
              "website":"https://competentauthoritydelhi.co.in/MonumentViewer.aspx?ID=243"
           }
        }
     },
     "bbox":[
        77.22013,
        28.59283,
        77.2205772,
        28.59322
     ]
  },
  {
     "type":"Feature",
     "geometry":{
        "type":"Point",
        "coordinates":[
           77.243142,
           28.656848
        ]
     },
     "properties":{
        "id":"way/223903623",
        "gid":"openstreetmap:venue:way/223903623",
        "layer":"venue",
        "source":"openstreetmap",
        "source_id":"way/223903623",
        "name":"Moti Masjid",
        "distance":11537.726,
        "accuracy":"point",
        "label":"Moti Masjid",
        "addendum":{
           "osm":{
              "wikidata":"Q2991448",
              "wikipedia":"en:Moti Masjid (Red Fort)"
           }
        }
     },
     "bbox":[
        77.2430408,
        28.6567821,
        77.2432438,
        28.6569142
     ]
  },
  {
     "type":"Feature",
     "geometry":{
        "type":"Point",
        "coordinates":[
           77.244858,
           28.609725
        ]
     },
     "properties":{
        "id":"way/370569205",
        "gid":"openstreetmap:venue:way/370569205",
        "layer":"venue",
        "source":"openstreetmap",
        "source_id":"way/370569205",
        "name":"Qila-i-Khuna Mosque",
        "distance":11542.77,
        "accuracy":"point",
        "label":"Qila-i-Khuna Mosque",
        "addendum":{
           "osm":{
              "wikidata":"Q19895556",
              "wikipedia":"en:Qila-i-Kuhna Mosque"
           }
        }
     },
     "bbox":[
        77.2447118,
        28.6094848,
        77.2449185,
        28.6099584
     ]
  },
  {
     "type":"Feature",
     "geometry":{
        "type":"Point",
        "coordinates":[
           77.185215,
           28.52483
        ]
     },
     "properties":{
        "id":"way/818709477",
        "gid":"openstreetmap:venue:way/818709477",
        "layer":"venue",
        "source":"openstreetmap",
        "source_id":"way/818709477",
        "name":"Quwwatul Islam Masjid",
        "housenumber":"Qutub Minar Complex",
        "street":"Qutub Minar Complex Rd, Seth Sarai, Mehrauli, New Delhi",
        "postalcode":"110030",
        "distance":11553.362,
        "accuracy":"point",
        "label":"Quwwatul Islam Masjid",
        "addendum":{
           "osm":{
              "wikidata":"Q15059100"
           }
        }
     },
     "bbox":[
        77.1849355,
        28.5245927,
        77.1854451,
        28.5249933
     ]
  }
]
}

Expected behavior label should have more details than just storing only the name

See the response from Mapzen's Pelias live site:

{
  "geocoding": {
    "version": "0.2",
    "attribution": "https://geocode.earth/guidelines",
    "query": {
      "text": "hauz khas",
      "parser": "pelias",
      "parsed_text": {
        "subject": "hauz khas"
      },
      "size": 10,
      "layers": [
        "venue",
        "street",
        "intersection",
        "postalcode",
        "locality",
        "neighbourhood",
        "county",
        "localadmin",
        "region",
        "macrocounty",
        "country",
        "macroregion",
        "dependency",
        "borough",
        "macrohood",
        "marinearea",
        "disputed",
        "empire",
        "continent",
        "ocean"
      ],
      "private": false,
      "focus.point.lat": 40.72605,
      "focus.point.lon": -73.98065,
      "lang": {
        "name": "English",
        "iso6391": "en",
        "iso6393": "eng",
        "via": "header",
        "defaulted": false
      },
      "querySize": 20
    },
    "warnings": [
      "performance optimization: excluding 'address' layer"
    ],
    "engine": {
      "name": "Pelias",
      "author": "Mapzen",
      "version": "1.0"
    },
    "timestamp": 1674801481681
  },
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "geometry": {
        "type": "Point",
        "coordinates": [
          77.205045,
          28.543369
        ]
      },
      "properties": {
        "id": "node/2832938172",
        "gid": "openstreetmap:venue:node/2832938172",
        "layer": "venue",
        "source": "openstreetmap",
        "source_id": "node/2832938172",
        "country_code": "IN",
        "name": "Hauz Khas",
        "distance": 11772.749,
        "accuracy": "point",
        "country": "India",
        "country_gid": "whosonfirst:country:85632469",
        "country_a": "IND",
        "region": "Delhi",
        "region_gid": "whosonfirst:region:85672123",
        "region_a": "DL",
        "county": "South West Delhi",
        "county_gid": "whosonfirst:county:1108695773",
        "county_a": "SW",
        "locality": "Delhi",
        "locality_gid": "whosonfirst:locality:102030425",
        "neighbourhood": "Hauz Khas",
        "neighbourhood_gid": "whosonfirst:neighbourhood:85928641",
        "continent": "Asia",
        "continent_gid": "whosonfirst:continent:102191569",
        "label": "Hauz Khas, Delhi, India",
        "addendum": {
          "osm": {
            "wikidata": "Q5683002",
            "wikipedia": "en:Hauz Khas metro station",
            "operator": "Delhi Metro Rail Corporation"
          }
        }
      }
    },
    {
      "type": "Feature",
      "geometry": {
        "type": "Point",
        "coordinates": [
          77.201363,
          28.547644
        ]
      },
      "properties": {
        "id": "way/357222241",
        "gid": "openstreetmap:venue:way/357222241",
        "layer": "venue",
        "source": "openstreetmap",
        "source_id": "way/357222241",
        "country_code": "IN",
        "name": "Haveli Hauz Khas",
        "housenumber": "P5",
        "street": "Hauz Khas Enclave",
        "postalcode": "110016",
        "distance": 11772.172,
        "accuracy": "point",
        "country": "India",
        "country_gid": "whosonfirst:country:85632469",
        "country_a": "IND",
        "region": "Delhi",
        "region_gid": "whosonfirst:region:85672123",
        "region_a": "DL",
        "county": "South West Delhi",
        "county_gid": "whosonfirst:county:1108695773",
        "county_a": "SW",
        "locality": "Delhi",
        "locality_gid": "whosonfirst:locality:102030425",
        "neighbourhood": "Hauz Khas",
        "neighbourhood_gid": "whosonfirst:neighbourhood:85928641",
        "continent": "Asia",
        "continent_gid": "whosonfirst:continent:102191569",
        "label": "Haveli Hauz Khas, Delhi, India",
        "addendum": {
          "osm": {
            "wheelchair": "limited",
            "website": "http://www.havelibnb.com/",
            "phone": "+91-981155-1357"
          }
        }
      },
      "bbox": [
        77.2012793,
        28.547513,
        77.2014239,
        28.5477436
      ]
    },
    {
      "type": "Feature",
      "geometry": {
        "type": "Point",
        "coordinates": [
          77.199167,
          28.542745
        ]
      },
      "properties": {
        "id": "way/358778374",
        "gid": "openstreetmap:venue:way/358778374",
        "layer": "venue",
        "source": "openstreetmap",
        "source_id": "way/358778374",
        "country_code": "IN",
        "name": "IIT Hauz Khas",
        "postalcode": "110016",
        "distance": 11772.595,
        "accuracy": "point",
        "country": "India",
        "country_gid": "whosonfirst:country:85632469",
        "country_a": "IND",
        "region": "Delhi",
        "region_gid": "whosonfirst:region:85672123",
        "region_a": "DL",
        "county": "South West Delhi",
        "county_gid": "whosonfirst:county:1108695773",
        "county_a": "SW",
        "locality": "Delhi",
        "locality_gid": "whosonfirst:locality:102030425",
        "neighbourhood": "IIT Campus",
        "neighbourhood_gid": "whosonfirst:neighbourhood:85928621",
        "continent": "Asia",
        "continent_gid": "whosonfirst:continent:102191569",
        "label": "IIT Hauz Khas, Delhi, India"
      },
      "bbox": [
        77.1990247,
        28.5426511,
        77.1993059,
        28.5428395
      ]
    },
    {
      "type": "Feature",
      "geometry": {
        "type": "Point",
        "coordinates": [
          77.193866,
          28.554202
        ]
      },
      "properties": {
        "id": "node/1024907338",
        "gid": "openstreetmap:venue:node/1024907338",
        "layer": "venue",
        "source": "openstreetmap",
        "source_id": "node/1024907338",
        "country_code": "IN",
        "name": "Hauz Khas Medresse Mosque",
        "distance": 11771.218,
        "accuracy": "point",
        "country": "India",
        "country_gid": "whosonfirst:country:85632469",
        "country_a": "IND",
        "region": "Delhi",
        "region_gid": "whosonfirst:region:85672123",
        "region_a": "DL",
        "county": "South West Delhi",
        "county_gid": "whosonfirst:county:1108695773",
        "county_a": "SW",
        "locality": "Delhi",
        "locality_gid": "whosonfirst:locality:102030425",
        "neighbourhood": "Safdarjang Development Area",
        "neighbourhood_gid": "whosonfirst:neighbourhood:85928659",
        "continent": "Asia",
        "continent_gid": "whosonfirst:continent:102191569",
        "label": "Hauz Khas Medresse Mosque, Delhi, India"
      }
    },
    {
      "type": "Feature",
      "geometry": {
        "type": "Point",
        "coordinates": [
          77.19157,
          28.55306
        ]
      },
      "properties": {
        "id": "8230441",
        "gid": "geonames:locality:8230441",
        "layer": "locality",
        "source": "geonames",
        "source_id": "8230441",
        "country_code": "IN",
        "name": "Hauz Khas",
        "distance": 11771.251,
        "accuracy": "centroid",
        "country": "India",
        "country_gid": "whosonfirst:country:85632469",
        "country_a": "IND",
        "region": "Delhi",
        "region_gid": "whosonfirst:region:85672123",
        "region_a": "DL",
        "county": "South West Delhi",
        "county_gid": "whosonfirst:county:1108695773",
        "county_a": "SW",
        "locality": "Hauz Khas",
        "locality_gid": "geonames:locality:8230441",
        "continent": "Asia",
        "continent_gid": "whosonfirst:continent:102191569",
        "label": "Hauz Khas, DL, India",
        "addendum": {
          "geonames": {
            "feature_code": "PPL"
          }
        }
      }
    },
    {
      "type": "Feature",
      "geometry": {
        "type": "Point",
        "coordinates": [
          74.191,
          30.46431
        ]
      },
      "properties": {
        "id": "1309964007",
        "gid": "whosonfirst:locality:1309964007",
        "layer": "locality",
        "source": "whosonfirst",
        "source_id": "1309964007",
        "country_code": "IN",
        "name": "Hauz Khas",
        "distance": 11461.007,
        "accuracy": "centroid",
        "country": "India",
        "country_gid": "whosonfirst:country:85632469",
        "country_a": "IND",
        "region": "Punjab",
        "region_gid": "whosonfirst:region:85672195",
        "region_a": "PB",
        "county": "Firozpur",
        "county_gid": "whosonfirst:county:890508949",
        "locality": "Hauz Khas",
        "locality_gid": "whosonfirst:locality:1309964007",
        "continent": "Asia",
        "continent_gid": "whosonfirst:continent:102191569",
        "label": "Hauz Khas, PB, India",
        "addendum": {
          "concordances": {
            "gn:id": 10016299
          }
        }
      },
      "bbox": [
        74.171,
        30.44431,
        74.211,
        30.48431
      ]
    },
    {
      "type": "Feature",
      "geometry": {
        "type": "Point",
        "coordinates": [
          77.193775,
          28.553648
        ]
      },
      "properties": {
        "id": "way/88213597",
        "gid": "openstreetmap:venue:way/88213597",
        "layer": "venue",
        "source": "openstreetmap",
        "source_id": "way/88213597",
        "country_code": "IN",
        "name": "Hauz Khas Medresse North Wing",
        "distance": 11771.272,
        "accuracy": "point",
        "country": "India",
        "country_gid": "whosonfirst:country:85632469",
        "country_a": "IND",
        "region": "Delhi",
        "region_gid": "whosonfirst:region:85672123",
        "region_a": "DL",
        "county": "South West Delhi",
        "county_gid": "whosonfirst:county:1108695773",
        "county_a": "SW",
        "locality": "Delhi",
        "locality_gid": "whosonfirst:locality:102030425",
        "neighbourhood": "Safdarjang Development Area",
        "neighbourhood_gid": "whosonfirst:neighbourhood:85928659",
        "continent": "Asia",
        "continent_gid": "whosonfirst:continent:102191569",
        "label": "Hauz Khas Medresse North Wing, Delhi, India"
      },
      "bbox": [
        77.1935921,
        28.5532336,
        77.1939163,
        28.5540294
      ]
    },
    {
      "type": "Feature",
      "geometry": {
        "type": "Point",
        "coordinates": [
          77.206817,
          28.556359
        ]
      },
      "properties": {
        "id": "relation/3893543",
        "gid": "openstreetmap:venue:relation/3893543",
        "layer": "venue",
        "source": "openstreetmap",
        "source_id": "relation/3893543",
        "country_code": "IN",
        "name": "Hauz Khas Apartments",
        "distance": 11771.477,
        "accuracy": "point",
        "country": "India",
        "country_gid": "whosonfirst:country:85632469",
        "country_a": "IND",
        "region": "Delhi",
        "region_gid": "whosonfirst:region:85672123",
        "region_a": "DL",
        "county": "South West Delhi",
        "county_gid": "whosonfirst:county:1108695773",
        "county_a": "SW",
        "locality": "Delhi",
        "locality_gid": "whosonfirst:locality:102030425",
        "neighbourhood": "Hauz Khas",
        "neighbourhood_gid": "whosonfirst:neighbourhood:85928641",
        "continent": "Asia",
        "continent_gid": "whosonfirst:continent:102191569",
        "label": "Hauz Khas Apartments, Delhi, India"
      },
      "bbox": [
        77.2061027,
        28.5542491,
        77.2091751,
        28.5564263
      ]
    }
  ],
  "bbox": [
    74.171,
    28.5426511,
    77.2091751,
    30.48431
  ]
}

Environment (please complete the following information): Docker

Pastebin/Screenshots pelias.json config that I'm using:

{
  "logger": {
    "level": "info",
    "timestamp": false
  },
  "esclient": {
    "apiVersion": "7.5",
    "hosts": [
      { "host": "elasticsearch" }
    ]
  },
  "elasticsearch": {
    "settings": {
      "index": {
        "refresh_interval": "10s",
        "number_of_replicas": "0",
        "number_of_shards": "1"
      }
    }
  },
  "acceptance-tests": {
    "endpoints": {
      "docker": "http://api:4000/v1/"
    }
  },
  "api": {
    "services": {
      "placeholder": { "url": "http://placeholder:4100" },
      "pip": { "url": "http://pip:4200" },
      "interpolation": { "url": "http://interpolation:4300" },
      "libpostal": { "url": "http://libpostal:4400" }
    },
    "defaultParameters": {
      "focus.point.lat": 45.52,
      "focus.point.lon": -122.67
    },
    "autocomplete": {
      "exclude_address_length": 2
    }
  },
  "imports": {
    "adminLookup": {
      "enabled": true
    },
    "blacklist": {
      "files": [
        "/data/blacklist/osm.txt"
      ]
    },
    "csv": {
      "datapath": "/data/csv",
      "files": [
        "example.csv"
      ],
      "download": [
        "https://raw.githubusercontent.com/pelias/csv-importer/master/data/example.csv"
      ]
    },
    "geonames": {
      "datapath": "/data/geonames",
      "countryCode": "ALL"
    },
    "openstreetmap": {
      "download": [
        { "sourceURL": "can not be empty" }
      ],
      "leveldbpath": "/tmp",
      "importVenues": true,
      "datapath": "/data/openstreetmap",
      "import": [
        {"filename": "ncr-zone-merged.osm.pbf"}
      ]
    },
    "openaddresses": {
      "datapath": "/data/openaddresses",
      "files": [
      ]
    },
    "polyline": {
      "datapath": "/data/polylines",
      "files": [ "extract.0sv" ]
    },
    "whosonfirst": {
      "datapath": "/data/whosonfirst",
      "importPostalcodes": true,
      "countryCode": "IND",
      "importPlace": [
      ]
    },
    "transit": {
      "datapath": "/data/transit",
      "feeds": [
      ]
    },
    "interpolation": {
      "download": {
        "tiger": {
          "datapath": "/data/tiger",
          "states": [
          ]
        }
      }
    }
  }
}

Additional context My only guess is that either it is a bug or I also need to import WOF data.

missinglink commented 1 year ago

Hi @theoway, you said you cloned the Portland Metro project and then it returned results in India such as 'Bara Gumbad tomb..'?

theoway commented 1 year ago

@missinglink I used Portland metro project as reference. I then imported my own dataset.

My config is attached in the issue (pelias.json)

missinglink commented 1 year ago

It looks like you're missing the administrative hierarchy information, eg. City, County etc.

In Pelias this information is added for every place by performing a point-in-polygon lookup against the whosonfirst data.

That whosonfirst data is downloaded during the download step and the index is generated in the prepare step.

I suspect that you're importing places for India but the administrative data for India isn't available.

theoway commented 1 year ago

Thanks! That makes sense. As you can see from the config, I'm only using OSM dataset.

So, that means I need to import WOF data as well for India.

Could you please give me notes on how I can download relevant WOF data?

PS: I'm still confused on how Nominatim is able to serve administrative data as well, considering both of my Pelias and Nominatim setup used the exact same OSM dataset, and nothing else.

missinglink commented 1 year ago

We don't use the administrative hierarchy from OSM which nominatim does, so they only need one data file, we combine multiple data sources, and use whosonfirst for the administrative hierarchy.

You don't strictly need to 'import' whosonfirst into Pelias, doing so will make each city, state etc searchable by adding a document to the elasticsearch index.

But you will certainly need to download and prepare the correct whosonfirst data so that when you import OSM each record is assigned administrative information.

theoway commented 1 year ago

@missinglink Thanks so much! This has helped a lot.