pip-service do not returned all whosonfirst data

[geopamplona]

Hey team!

I was using your awesome geocoding engine when I noticed something interesting. Let me tell you more about it.

---

Here's what I did :innocent:

Iam using docker image pelias/pip-service:master-2019-08-02-3e321b769b4722edc18e84eb0b852f27cefcd5a3

I start service with:

npm run download
npm start

Donwload all files correctly in /mnt/pelias..etc Then start service, load all files correctly and start.

I try some coords with curl http://localhost:4200/0.6228303909301759/41.614511872342064

---

Here's what I got :scream_cat:

{
   "continent":[
      {
         "id":102191581,
         "name":"Europe",
         "centroid":{
            "lat":55.924452,
            "lon":28.09812
         },
         "bounding_box":"-24.539906,34.815009,69.033946,81.85871"
      }
   ],
   "country":[
      {
         "id":85633129,
         "name":"Spain",
         "abbr":"ESP",
         "centroid":{
            "lat":40.228091,
            "lon":-3.646342
         },
         "bounding_box":"-18.16077363,27.63783897,4.32773866,43.7913417"
      }
   ],
   "region":[
      {
         "id":85682769,
         "name":"Lleida",
         "abbr":"CT",
         "centroid":{
            "lat":42.043843,
            "lon":1.047634
         },
         "bounding_box":"0.32003648,41.27412541,1.85505769,42.86145031"
      }
   ]
}

---

Here's what I was expecting :sparkles:

I expected information about the locality

The continent and the region is correct (if understand region as a province); in the case of Spain there is an entity upper to the province that would be Catalonia (autonomous community that can group several provinces).

It failed in the autonomous community of Catalonia and locality data.

At the beginning I had tried to download the data of the repo https://github.com/whosonfirst-data/whosonfirst-data-admin-es but neither did I work well with the global data, but it also does not work.

Is it possible that whosonfirst does not have this data? What data source could have it?

Thanks !

[missinglink]

Can you please paste the imports.whosonfirst section of your pelias.json?

[geopamplona]

I have that section empty

I asume default directory and config for whosonfirst importer. I am using default config

"imports": {
        "adminLookup": {
            "enabled": true
        },
        "openstreetmap": {
            "leveldbpath": "/tmp",
            "datapath": "/code",
            "importVenues": false,
            "import": [{
                    "filename": "osm.pbf",
                    "importVenues": false
                }]
        },
        "services": {
            "pip": {
                "url": "http://pip:4200"
            }
        }
    }

Thanks @missinglink

[missinglink]

ok, if you don't specify imports.whosonfirst it will indeed use the defaults listed at the bottom of https://github.com/pelias/config/blob/master/config/defaults.json

So, in that case, you should be able to browse the directory /mnt/pelias/whosonfirst to check which data you have. There are three potential causes for your problem:

1. The place you're seeking is present in the WhosOnFirst data but not on your HDD for some reason
2. The place you're seeking is not in WhosOnFirst, in which case you'll need to file an issue with them to discuss why and try to find a new source of data
3. There is something wrong with either your configuration or our code

Regarding issue 2. there is an existing issue to update this data from ICGC.

I hope that is useful, I can only really help you more with 3. if there is a bug in our code. If you find a resolution to this issue, please hit 'Close Issue' to indicate it has been resolved.

[geopamplona]

I think the problem could be related to the first point:

• The place you are looking for is present in the WhosOnFirst data but not in your HDD for some reason
• There is something wrong with your configuration or our code

I don't think it's a code problem, but it's hard for me to understand what happens

the place is presented in whosonfirst; In this case I do the same search and find it correctly https://pelias.github.io/compare/#/v1/search%3Ftext=L%C3%A9rida,%20Espa%C3%B1a

To solve the first point I use the command npm run download

it is assumed that with this command I download the complete whosonfirst database and the charge with pip as a service when start service with npm run start

Checking this last point I see that for example in file "/mnt/pelias/whosonfirst/meta/whosonfirst-data-macroregion-latest.csv" only has 26 rows; This would be correct !?

edit:

i search Catalonia or Cataluña or Catalunya in all bundles from whoson first meta csv but i dont find anything about Catalonia like macroregion or similar.. iam confusing about data

[orangejulius]

I think the underlying problem here is the same as another issue, https://github.com/pelias/docker/issues/141#issuecomment-558172560.

Some of the Who's on First bundles are corrupted. For now, you can set sqlite: true in your `pelias.json (as described in the above issue). We'll be making the SQLite files the default soon as this is the direction the projects going in. Once that config value is set, you should be able to re-download and import and see correct results.

[geopamplona]

This problem is solved using sqlite database.

My new config:

        "whosonfirst": {
            "sqlite": true,
            "datapath": "/mnt/pelias/whosonfirst",
            "importVenues": false,
            "importPostalcodes": true
        },

I download whosonfirst-data-admin-es-latest.db.bz2; decompress it in /mnt/pelias/whosonfirst/sqlite/whosonfirst-data-latest.db

and start service npm run start

Do request curl http://localhost:4200/0.6228303909301759/41.614511872342064 Response is more complete than after:

{
   "country":[
      {
         "id":85633129,
         "name":"Spain",
         "abbr":"ESP",
         "centroid":{
            "lat":40.228091,
            "lon":-3.646342
         },
         "bounding_box":"-18.16077363,27.63783897,4.32773866,43.7913417"
      }
   ],
   "localadmin":[
      {
         "id":404337851,
         "name":"Lleida",
         "centroid":{
            "lat":41.631316,
            "lon":0.581205
         },
         "bounding_box":"0.36355709267,41.5101224507,0.742513433361,41.7396155375"
      }
   ],
   "locality":[
      {
         "id":101748359,
         "name":"Lérida",
         "centroid":{
            "lat":41.620615,
            "lon":0.621512
         },
         "bounding_box":"0.5902115,41.5966513431,0.649371782253,41.6397745729"
      }
   ],
   "macroregion":[
      {
         "id":404227379,
         "name":"Catalonia",
         "centroid":{
            "lat":41.798469,
            "lon":1.528745
         },
         "bounding_box":"0.15918115,40.52304652,3.33255381,42.86145031"
      }
   ],
   "region":[
      {
         "id":85682769,
         "name":"Lleida",
         "abbr":"CT",
         "centroid":{
            "lat":42.043952,
            "lon":1.047755
         },
         "bounding_box":"0.319930552327,41.2742123194,1.85553189125,42.8614503147"
      }
   ]
}

thanks @orangejulius !