Vagrant-pelias - error loading data when admin lookup option is enabled

[pawelchmara]

Hi,

when admin lookup option is enabled in pelias configuration file then errors occur when osm data is loaded.

Error logs from /opt/pelias/logs:

Error: ENOENT, open '/mnt/pelias/whosonfirst/meta/wof-localadmin-latest.csv'
    at Error (native)
events.js:85
      throw er; // Unhandled 'error' event
            ^
Error: ENOENT, open '/mnt/pelias/whosonfirst/meta/wof-macrocounty-latest.csv'
    at Error (native)
events.js:85
      throw er; // Unhandled 'error' event
            ^
Error: ENOENT, open '/mnt/pelias/whosonfirst/meta/wof-disputed-latest.csv'
    at Error (native)
events.js:85
      throw er; // Unhandled 'error' event
            ^
Error: ENOENT, open '/mnt/pelias/whosonfirst/meta/wof-county-latest.csv'
    at Error (native)
events.js:85
      throw er; // Unhandled 'error' event
            ^
Error: ENOENT, open '/mnt/pelias/whosonfirst/meta/wof-locality-latest.csv'
    at Error (native)
events.js:85
      throw er; // Unhandled 'error' event
            ^
Error: ENOENT, open '/mnt/pelias/whosonfirst/meta/wof-dependency-latest.csv'
    at Error (native)
events.js:85
      throw er; // Unhandled 'error' event
            ^
Error: ENOENT, open '/mnt/pelias/whosonfirst/meta/wof-borough-latest.csv'
    at Error (native)
events.js:85
      throw er; // Unhandled 'error' event
            ^
Error: ENOENT, open '/mnt/pelias/whosonfirst/meta/wof-macroregion-latest.csv'
    at Error (native)
events.js:85
      throw er; // Unhandled 'error' event
            ^
Error: ENOENT, open '/mnt/pelias/whosonfirst/meta/wof-neighbourhood-latest.csv'
    at Error (native)
events.js:85
      throw er; // Unhandled 'error' event
            ^
Error: ENOENT, open '/mnt/pelias/whosonfirst/meta/wof-country-latest.csv'
    at Error (native)
events.js:85
      throw er; // Unhandled 'error' event
            ^
Error: ENOENT, open '/mnt/pelias/whosonfirst/meta/wof-region-latest.csv'
    at Error (native)

curl -s 'http://localhost:9200/_cluster/health?pretty=true'
{
  "cluster_name" : "elasticsearch",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 1,
  "active_shards" : 1,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0
}

curl -s 'http://localhost:9200'
{
  "status" : 200,
  "name" : "pelias",
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "1.7.3",
    "build_hash" : "05d4530971ef0ea46d0f4fa6ee64dbc8df659682",
    "build_timestamp" : "2015-10-15T09:14:17Z",
    "build_snapshot" : false,
    "lucene_version" : "4.10.4"
  },
  "tagline" : "You Know, for Search"
}

Vagrant configuration file:

Vagrant.configure('2') do |config|
  config.vm.provision :chef_solo do |chef|
    chef.cookbooks_path = 'cookbooks'

    chef.json = {
      'pelias' => {
        'schema' => {
          'create_index' => true
        },
        'openaddresses' => {
          'index_data' => false,
          'data_dir' => '/opt/pelias/data/openaddresses',
          'data_files' => [
            'dolnoslaskie.csv'
          ]
        },
        'geonames' => {
          'index_data' => false,
          'alpha2_country_codes' => [
            'GB'
          ]
        },
        'osm' => {
          'index_data' => true,
          'admin_lookup' => true,
          'extracts' => { http://download.geofabrik.de/europe/poland/dolnoslaskie-latest.osm.pbf'
          }
        }
      }
    }

    chef.run_list = [
      'recipe[pelias::default]'
    ]
  end
end

Thanks for help!

[missinglink]

hi @pawelchmara

the admin-lookup module adds country region neighborhood etc. info to documents, it's optional and was a little hard to bundle in the vagrant image (for reasons explained below)

in order to enable it you will need to download the data files, they are, as always free and open :)

you'll need to clone this repo https://github.com/whosonfirst/whosonfirst-data somewhere on your machine, it's 25GB, so consider that when you download it and possibly get a hot drink while you wait.

in order to tell pelias where those files live you can update your ~/pelias.json to override the default values:

  "imports": {
    "whosonfirst": {
      "adminLookup": true,
      "datapath": "/data/whosonfirst-data"
    },

that's it, it should work now and you can set "adminLookup": true for all the other importers to take advantage of it.

It's a difficult thing to include in the vagrant image due to it's size on disk and the RAM requirement to run, previous versions used up to 8GB of RAM to cache the geometries, I'm not sure how much is required for the current version but it should be much less.

we actually have a new version which requires less than 1GB of disk and almost no RAM, so once we get that polished we hope to include that within the image itself.

hope that helps, let me know how you get on.

[pawelchmara]

Thank you for the detailed answer.

I would like to ask about one more think but first I show example of autocomplete query:

{  
   "geocoding":{  
      "version":"0.1",
      "attribution":"https://search.mapzen.com/v1/attribution",
      "query":{  
         "text":"Wrocław",
         "tokens":[  
            "Wrocław"
         ],
         "size":10,
         "private":false
      },
      "warnings":[  
         "out-of-range integer 'size', using MIN_SIZE"
      ],
      "engine":{  
         "name":"Pelias",
         "author":"Mapzen",
         "version":"1.0"
      },
      "timestamp":1468411526893
   },
   "type":"FeatureCollection",
   "features":[  
      {  
         "type":"Feature",
         "geometry":{  
            "type":"Point",
            "coordinates":[  
               17.023978,
               51.097349
            ]
         },
         "properties":{  
            "id":"101752181",
            "gid":"whosonfirst:locality:101752181",
            "layer":"locality",
            "source":"whosonfirst",
            "source_id":"101752181",
            "name":"Wrocław",
            "confidence":0.951,
            "country":"Poland",
            "country_gid":"whosonfirst:country:85633723",
            "country_a":"POL",
            "region":"Dolnośląskie",
            "region_gid":"whosonfirst:region:85687301",
            "county":"M. WrocÅaw",
            "county_gid":"whosonfirst:county:102079661",
            "locality":"Wrocław",
            "locality_gid":"whosonfirst:locality:101752181",
            "label":"Wrocław, Poland"
         },
         "bbox":[  
            16.844077,
            51.0381465722,
            17.1654155,
            51.203595
         ]
      },

In the above feature there is information about country, region and locality. Install and run Whosonfirst-data project is the only way to get this information on my own server?

[orangejulius]

That's correct. If you want the fields like locality, region, region_gid, etc to show up, you have to download the whosonfirst-data repo and set up admin lookup. Let us know if you need any more help. Our installation docs and admin-lookup readme have more info that may be useful.

[pawelchmara]

Now all is clear to me. Thanks.

Could you also look at this issue: https://github.com/pelias/vagrant/issues/51

[dianashk]

The vagrant repository has been deprecated in favor of pelias/dockerfiles.