import locations from openstreetmap

[konektaz]

Currently we have location in the database that most show the lat/long and the hospital name only. There are 'none' placeholders where no data has been imported for the location address. This is because the initial import did not query/import that data. This issue looks to fill the location data fields http://www.konekta.info/location/add/ and clean out any duplicates.

Here is the method suggested:

Harvesting location data

First you go to: http://overpass-api.de/query_form.html There are 2 boxes (textarea) where you need to enter your XML query.

Example query looks like this one: https://github.com/konekta/where-is-when-is/blob/master/world/fixtures/south_africa_query.xml

put it to second box and select "to OpenLayers auto-centered overlay" and click convert. It'll show you the map of locations.

The example query searches for hospital or clinic in defined coordinates.

This is done automatically by this script: https://github.com/konekta/where-is-when-is/blob/master/world/management/commands/osm_import.py

Reverse geocoding

Now the second bit is to reverse goecode the retrieved coordinates against OSM. It's much easier as you don't need to create XML query but just create standard URL: http://wiki.openstreetmap.org/wiki/Nominatim#Reverse_Geocoding_.2F_Address_lookup

Basically new script needs to loop through all the locations in database that haven't address data (but they have coordinates) and fetch the data from URL: http://nominatim.openstreetmap.org/reverse?format=json&lat=52.5487429714954&lon=-1.81602098644987&zoom=18&addressdetails=1

and save back to database.

Tasks

• [x] Create a script to reverse geocode existing Location objects
• [x] Create a script to find and remove duplicate objects Location objects
• [x] Adjust the existing script for harvesting location data (world.commands.osm_import) so that it does not import duplicate objects

  Notes

• We should to find duplicate objects based on: external_id (which equates to osm_id) Name Address details
• The current osm_import script seems to allow duplicate items to be imported by simply saving the object without an external_id which is set to be unique.

[grvhi]

The xml returned from the first box is actually more useful for a function (as it returns raw xml data)

[konektaz]

there are duplicate locations in the DB. When we do the lat/long part of the task which I believe links the location name the the lat/long we need to make sutre there are no duplicates. http://www.konekta.info/south-africa/ http://www.konekta.info/location/crescent-clinic/

[grvhi]

@konekta @timlinux

My plan with finding duplicates is this:

Create a function which performs the following tasks:

1. Find all objects which have no external_id.
   • If there are no address details, delete it.
   • If there are address details, check if there is a duplicate entry with an external_id and merge / delete
2. Find objects with exactly the same name
   • Merge them or delete the one with no details.
3. Perhaps find matching address details (where both have no external_id)?

Feel free to change my plans!

[grvhi]

De-Duplication

In the interest of time-saving, the problem of de-duplication will be solved by deleting all existing Location data and re-importing all locations from OSM. At a later date, we will need to implement a system to prevent users from manually adding already existing locations - #134

Other Tasks

• The osm_import script has been improved and will no longer import duplicates (based on the node ID returned by OSM)
• A new addr_import script has been written to query nominatum for address data for all objects in the Location table. The script accepts Location IDs (not external_id) as args if you wish to get data for a specific object, otherwise it will get data for all objects in the Location table. The mapping of address details may need some attention as the key values from nominatum are not necessarily consistent.

CC

@konekta @timlinux