Missing Address Layer - Githubissues

DerSchwarzeJugo commented 5 months ago

Describe the bug Address Layer seems to be missing when a parameter is specifically set for it (layers=address) while requesting /search or /autocomplete endpoint. Receiving the following error code 400 "Bad Request":

    "errors": [
      "'address' is an invalid layers parameter. Valid options: coarse,postalcode,locality,neighbourhood,localadmin,county,region,macrohood,macrocounty,borough,macroregion,country,disputed,marinearea,empire,dependency,continent"
    ],

When not setting the "layers" paramete while requestingr, it is returning a correct respone where the shown feauture is from source "openstreetmap" and layer type "address".

--> Therefore, I am unable to use autocomplete function for addresses only. NOTE: I did a custom build for switzerland before, and it did work correctly with the same type of setup (the only difference, that I was using the openaddress importer there as well).

Steps to Reproduce Following the default docker setup guide with previous custom europe build setup. Using only the importers geonames, openstreetmap, polylines (for that the pbf was to big, but that should not have any influnce in this issue), whosonfirst.

Providing my files in section below.

Expected behavior I should be able to provide the value "address" for parameter "layer" and receive a Response with code 200

Environment (please complete the following information): OS: Linux octa-pelias-001 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux Docker: Docker version 24.0.7, build afdd53b Docker Compose: Docker Compose version v2.21.0

Pastebin/Screenshots

Additional context docker-compose.yml

version: '3'
networks:
  default:
    driver: bridge
services:
  libpostal:
    image: pelias/libpostal-service
    container_name: pelias_libpostal
    user: "${DOCKER_USER}"
    restart: always
    ports: [ "127.0.0.1:4400:4400" ]
  schema:
    image: pelias/schema:master
    container_name: pelias_schema
    user: "${DOCKER_USER}"
    volumes:
      - "./pelias.json:/code/pelias.json"
  api:
    image: pelias/api:master
    container_name: pelias_api
    user: "${DOCKER_USER}"
    restart: always
    environment: [ "PORT=4000" ]
    ports: [ "0.0.0.0:443:4000" ]
    volumes:
      - "./pelias.json:/code/pelias.json"
  placeholder:
    image: pelias/placeholder:master
    container_name: pelias_placeholder
    user: "${DOCKER_USER}"
    restart: always
    environment: [ "PORT=4100" ]
    ports: [ "127.0.0.1:4100:4100" ]
    volumes:
      - "./pelias.json:/code/pelias.json"
      - "${DATA_DIR}:/data"
      - "./blacklist/:/data/blacklist"
  whosonfirst:
    image: pelias/whosonfirst:master
    container_name: pelias_whosonfirst
    user: "${DOCKER_USER}"
    volumes:
      - "./pelias.json:/code/pelias.json"
      - "${DATA_DIR}:/data"
      - "./blacklist/:/data/blacklist"
  openstreetmap:
    image: pelias/openstreetmap:master
    container_name: pelias_openstreetmap
    user: "${DOCKER_USER}"
    volumes:
      - "./pelias.json:/code/pelias.json"
      - "${DATA_DIR}:/data"
      - "./blacklist/:/data/blacklist"
  openaddresses:
    image: pelias/openaddresses:master
    container_name: pelias_openaddresses
    user: "${DOCKER_USER}"
    volumes:
      - "./pelias.json:/code/pelias.json"
      - "${DATA_DIR}:/data"
      - "./blacklist/:/data/blacklist"
  transit:
    image: pelias/transit:master
    container_name: pelias_transit
    user: "${DOCKER_USER}"
    volumes:
      - "./pelias.json:/code/pelias.json"
      - "${DATA_DIR}:/data"
  csv-importer:
    image: pelias/csv-importer:master
    container_name: pelias_csv_importer
    user: "${DOCKER_USER}"
    volumes:
      - "./pelias.json:/code/pelias.json"
      - "${DATA_DIR}:/data"
      - "./blacklist/:/data/blacklist"
  polylines:
    image: pelias/polylines:master
    container_name: pelias_polylines
    user: "${DOCKER_USER}"
    volumes:
      - "./pelias.json:/code/pelias.json"
      - "${DATA_DIR}:/data"
  interpolation:
    image: pelias/interpolation:master
    container_name: pelias_interpolation
    user: "${DOCKER_USER}"
    restart: always
    environment: [ "PORT=4300" ]
    ports: [ "127.0.0.1:4300:4300" ]
    volumes:
      - "./pelias.json:/code/pelias.json"
      - "${DATA_DIR}:/data"
  pip:
    image: pelias/pip-service:master
    container_name: pelias_pip-service
    user: "${DOCKER_USER}"
    restart: always
    environment: [ "PORT=4200" ]
    ports: [ "127.0.0.1:4200:4200" ]
    volumes:
      - "./pelias.json:/code/pelias.json"
      - "${DATA_DIR}:/data"
  elasticsearch:
    image: pelias/elasticsearch:7.16.1
    container_name: pelias_elasticsearch
    user: "${DOCKER_USER}"
    restart: always
    ports: [ "127.0.0.1:9200:9200", "127.0.0.1:9300:9300" ]
    volumes:
      - "${DATA_DIR}/elasticsearch:/usr/share/elasticsearch/data"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    cap_add: [ "IPC_LOCK" ]
  fuzzy-tester:
    image: pelias/fuzzy-tester:master
    container_name: pelias_fuzzy_tester
    user: "${DOCKER_USER}"
    restart: "no"
    command: "--help"
    volumes:
      - "./pelias.json:/code/pelias.json"
      - "./test_cases:/code/pelias/fuzzy-tester/test_cases"

pelias.json

{
  "logger": {
    "level": "debug",
    "timestamp": false
  },
  "esclient": {
    "apiVersion": "7.5",
    "hosts": [
      { "host": "elasticsearch" }
    ]
  },
  "elasticsearch": {
    "settings": {
      "index": {
        "refresh_interval": "10s",
        "number_of_replicas": "0",
        "number_of_shards": "1"
      }
    }
  },
  "api": {
    "services": {
      "pip": { "url": "http://pip:4200" },
      "libpostal": { "url": "http://libpostal:4400" },
      "placeholder": { "url": "http://placeholder:4100" },
      "interpolation": { "url": "http://interpolation:4300" }
    }
  },
  "imports": {
    "adminLookup": {
      "enabled": true
    },
    "geonames": {
      "datapath": "/data/geonames",
      "countryCode": "EU"
    },
    "openstreetmap": {
      "download": [
        { "sourceURL": "http://download.geofabrik.de/europe-latest.osm.pbf" }
      ],
      "leveldbpath": "/tmp",
      "datapath": "/data/openstreetmap",
      "import": [{
        "filename": "europe-latest.osm.pbf"
      }]
    },
    "polyline": {
      "datapath": "/data/polylines",
      "files": [ "extract.0sv" ]
    },
    "whosonfirst": {
      "datapath": "/data/whosonfirst",
      "importPostalcodes": true,
      "importPlace": [ "102191581" ]
    }
  }
}

Working Request /search https:///v1/search/structured?address=bahnhofstrasse 1&postalcode=8000&locality=Zurich&country=CH

{
    "geocoding": {
        "version": "0.2",
        "attribution": "<Removed for Privacy>",
        "query": {
            "parsed_text": {
                "city": "zurich",
                "postalcode": "8000",
                "country": "CHE",
                "housenumber": "1",
                "street": "bahnhofstrasse"
            },
            "size": 10,
            "private": false,
            "lang": {
                "name": "German",
                "iso6391": "de",
                "iso6393": "deu",
                "via": "header",
                "defaulted": false
            },
            "querySize": 20
        },
        "engine": {
            "name": "Pelias",
            "author": "Mapzen",
            "version": "1.0"
        },
        "timestamp": 1706523427809
    },
    "type": "FeatureCollection",
    "features": [
        {
            "type": "Feature",
            "geometry": {
                "type": "Point",
                "coordinates": [
                    8.539791,
                    47.367324
                ]
            },
            "properties": {
                "id": "way/89971420",
                "gid": "openstreetmap:address:way/89971420",
                "layer": "address",
                "source": "openstreetmap",
                "source_id": "way/89971420",
                "country_code": "CH",
                "name": "Bahnhofstrasse 1",
                "housenumber": "1",
                "street": "Bahnhofstrasse",
                "postalcode": "8001",
                "confidence": 1,
                "match_type": "exact",
                "accuracy": "point",
                "country": "Schweiz",
                "country_gid": "whosonfirst:country:85633051",
                "country_a": "CHE",
                "region": "Zürich",
                "region_gid": "whosonfirst:region:85682309",
                "region_a": "ZH",
                "county": "Zürich",
                "county_gid": "whosonfirst:county:102063071",
                "localadmin": "Zürich",
                "localadmin_gid": "whosonfirst:localadmin:1394254167",
                "locality": "Zürich",
                "locality_gid": "whosonfirst:locality:101748463",
                "neighbourhood": "Enge",
                "neighbourhood_gid": "whosonfirst:neighbourhood:85799955",
                "continent": "Europa",
                "continent_gid": "whosonfirst:continent:102191581",
                "label": "Bahnhofstrasse 1, Zürich, Schweiz"
            }
        }
    ],
    "bbox": [
        8.539791,
        47.367324,
        8.539791,
        47.367324
    ]
}

Failing Request /search https:///v1/search/structured?address=bahnhofstrasse 1&postalcode=8000&locality=Zurich&country=CH&layers=address

{
    "geocoding": {
        "version": "0.2",
        "attribution": "<Removed for Privacy>",
        "query": {
            "parsed_text": {
                "address": "bahnhofstrasse 1",
                "city": "zurich",
                "postalcode": "8000",
                "country": "CHE"
            },
            "size": 10,
            "private": false,
            "lang": {
                "name": "German",
                "iso6391": "de",
                "iso6393": "deu",
                "via": "header",
                "defaulted": false
            },
            "querySize": 20
        },
        "errors": [
            "'address' is an invalid layers parameter. Valid options: coarse,postalcode,locality,neighbourhood,localadmin,county,region,macrohood,macrocounty,borough,macroregion,country,disputed,marinearea,empire,dependency,continent"
        ],
        "engine": {
            "name": "Pelias",
            "author": "Mapzen",
            "version": "1.0"
        },
        "timestamp": 1706523555145
    },
    "type": "FeatureCollection",
    "features": []
}

Working Request /autocomplete https:///v1/autocomplete?text=bahnhofstrasse&focus.point.lat=47.295006918341564&focus.point.lon=8.24202060699463&size=25

{
    "geocoding": {
        "version": "0.2",
        "attribution": "<Removed for Privacy>",
        "query": {
            "text": "bahnhofstrasse",
            "parser": "pelias",
            "parsed_text": {
                "subject": "bahnhofstrasse",
                "street": "bahnhofstrasse"
            },
            "size": 25,
            "layers": [
                "postalcode",
                "locality",
                "neighbourhood",
                "localadmin",
                "county",
                "region",
                "macrohood",
                "macrocounty",
                "borough",
                "macroregion",
                "country",
                "disputed",
                "marinearea",
                "empire",
                "dependency",
                "continent"
            ],
            "private": false,
            "focus.point.lat": 47.295006918341564,
            "focus.point.lon": 8.24202060699463,
            "lang": {
                "name": "German",
                "iso6391": "de",
                "iso6393": "deu",
                "via": "header",
                "defaulted": false
            },
            "querySize": 50
        },
        "warnings": [
            "performance optimization: excluding 'address' layer"
        ],
        "engine": {
            "name": "Pelias",
            "author": "Mapzen",
            "version": "1.0"
        },
        "timestamp": 1706523629678
    },
    "type": "FeatureCollection",
    "features": [
        {
            "type": "Feature",
            "geometry": {
                "type": "Point",
                "coordinates": [
                    8.61935,
                    47.39778
                ]
            },
            "properties": {
                "id": "1126062871",
                "gid": "whosonfirst:neighbourhood:1126062871",
                "layer": "neighbourhood",
                "source": "whosonfirst",
                "source_id": "1126062871",
                "country_code": "CH",
                "name": "Bahnhofstrasse",
                "distance": 30.674,
                "accuracy": "centroid",
                "country": "Schweiz",
                "country_gid": "whosonfirst:country:85633051",
                "country_a": "CHE",
                "region": "Zürich",
                "region_gid": "whosonfirst:region:85682309",
                "region_a": "ZH",
                "county": "Uster District",
                "county_gid": "whosonfirst:county:102062933",
                "localadmin": "Dübendorf",
                "localadmin_gid": "whosonfirst:localadmin:404569055",
                "locality": "Dübendorf",
                "locality_gid": "whosonfirst:locality:1125826975",
                "neighbourhood": "Bahnhofstrasse",
                "neighbourhood_gid": "whosonfirst:neighbourhood:1126062871",
                "continent": "Europa",
                "continent_gid": "whosonfirst:continent:102191581",
                "label": "Bahnhofstrasse, Dübendorf, ZH, Schweiz",
                "addendum": {
                    "concordances": {
                        "gn:id": 6294303,
                        "qs_pg:id": 620404,
                        "qs:id": 620404
                    }
                }
            },
            "bbox": [
                8.61435,
                47.39278,
                8.62435,
                47.40278
            ]
        }
    ],
    "bbox": [
        8.61435,
        47.39278,
        8.62435,
        47.40278
    ]
}

Failing Request /autocomplete https:///v1/autocomplete?text=bahnhofstrasse&focus.point.lat=47.295006918341564&focus.point.lon=8.24202060699463&layers=address&size=25

{
    "geocoding": {
        "version": "0.2",
        "attribution": "<Removed for Privacy>",
        "query": {
            "text": "bahnhofstrasse",
            "parser": "pelias",
            "parsed_text": {
                "subject": "bahnhofstrasse",
                "street": "bahnhofstrasse"
            },
            "size": 25,
            "private": false,
            "focus.point.lat": 47.295006918341564,
            "focus.point.lon": 8.24202060699463,
            "lang": {
                "name": "German",
                "iso6391": "de",
                "iso6393": "deu",
                "via": "header",
                "defaulted": false
            },
            "querySize": 50
        },
        "errors": [
            "'address' is an invalid layers parameter. Valid options: coarse,postalcode,locality,neighbourhood,localadmin,county,region,macrohood,macrocounty,borough,macroregion,country,disputed,marinearea,empire,dependency,continent"
        ],
        "engine": {
            "name": "Pelias",
            "author": "Mapzen",
            "version": "1.0"
        },
        "timestamp": 1706523758118
    },
    "type": "FeatureCollection",
    "features": []
}

References

I have already tried to get some help in the forum at https://app.gitter.im/#/room/#pelias_pelias:gitter.im

missinglink commented 5 months ago

hmm.. I suspect an uncommon series of events has caused this issue.

when the pelias/api server first starts up, it reads the canonical_sources & layers_by_source properties from your pelias.json, so by default the address layer is included .

in that config there is a boolean auto_discover: true which then tells the server to query elasticsearch to enumerate a list of the actual sources/layers in your data (this allows custom data to be discovered at runtime).

if you're using docker, the pelias elastic stats command executes the same query, which you can use to see what data was returned during that step.

after the response from elasticsearch the sources/layers are updated and if no address layer is found then it will be removed from the list of available layers.

since you mentioned that you do have data from the address layer in elasticsearch it might be some sort of race condition, restarting the pelias/api container (but not the pelias/elasticsearch) container should resolve the issue?

missinglink commented 5 months ago

alternatively setting auto_discover: false before restarting the pelias/api container will skip that discovery step, if you don't have custom sources or layers then that should work for you.

DerSchwarzeJugo commented 5 months ago

hmm.. I suspect an uncommon series of events has caused this issue.

when the pelias/api server first starts up, it reads the canonical_sources & layers_by_source properties from your pelias.json, so by default the address layer is included .

in that config there is a boolean auto_discover: true which then tells the server to query elasticsearch to enumerate a list of the actual sources/layers in your data (this allows custom data to be discovered at runtime).

if you're using docker, the pelias elastic stats command executes the same query, which you can use to see what data was returned during that step.

after the response from elasticsearch the sources/layers are updated and if no address layer is found then it will be removed from the list of available layers.

since you mentioned that you do have data from the address layer in elasticsearch it might be some sort of race condition, restarting the pelias/api container (but not the pelias/elasticsearch) container should resolve the issue?

Indeed a weird issue... restarting the pelias/api container did resolve the problem now. But I am pretty sure, i already tried that before and it did not help then. Anyways, thank you a lot for helping out so quickly.

pelias / docker

Missing Address Layer #339

References