Automate cities borders process by fetching them from OpenStreetMaps

[seddik]

Can't we automate the cities coordinates process by fetching the data directly from OpenStreetMaps, I suppose there is a way to do that.

I'll take a look at it and try to get that data

[seddik]

Found this solution

1- we get the Wilaya or commune

2- we get the exact name

3- we request the data from the Open Street Maps API

[seddik]

I'll create an API or a web page where you insert the exact name and it merges the geometry coordinates and gives a proper result that can be used in this repo.

[seddik]

Adding the wilaya/commune OpenStreetMaps ID is a better approach, when we add all Ids we can fetch data using a script.

[bensialih]

@seddik if this works... I think this is an even better solution than just doing things piecemeal. Let me know what you come up with.

One thing that we may need to factor is that we need to make sure that the geospatial coordinates are that of the wilaya. Can we get a POC to make sure this works and that we get a nice set of coordinates for a wilaya. Are you able to paste a set of coordinates so I can check

[seddik]

@bensialih it sure works I did some tests, we will generate data as soon as we fill all the OSM IDs, and in just one click 😁

[seddik]

@bensialih right now it is made as segments, I am looking for a way to merge them, we can't just put them together, I have created an issue for that #122

you can get coordinates from the api directly this for example for the OSMID temacine 6542943 and it gives you 4 segments, we need to convert it to a polygon. I'll handle that task #122 I did it before in some projects I think I have the algorithm, otherwise I'll recreate it.

http://overpass-api.de/api/interpreter?data=[out:json];(rel(6542943)[type=boundary];);out geom;

[bensialih]

@seddik I have had a look at the search api endpoint as well as the visual representation therefore.

Short of using the OSM ID 1279811 and just going through each wilaya in Algeria and writing down the ids for each one manually and then running a script like that to get the polygon, not sure how it can be down.

Maybe you can shed light on a better way, if not, I will start jotting down each id manually then using a script to hit their nominatim endpoint and go from there. Im hoping to get this done by the end of next weekend.

[bensialih]

I think I still hold the philosophy of move-quick-move-fast and try and get the first iteration of the wilayas done and then look at the sub-jurisdictions thereafter.

[seddik]

Sorry, but I really didn't understood your question ?!

[seddik]

As a start I did the full algeriacities.json, I found cities there so I started with that. this is why I didn't work on the wilayas first.

let's check if there is a wilayas file then we can inject -1 for all of them then we paste the osm ids.

but the json file there us cities, I don't think I saw any wilayas there

[seddik]

this is the format all along. we can create another file named wilayas, then another one named dairas. it should follow the same procedure.

you can create those files by a script or if you want create an issue, then whoever have time create them

[bensialih]

ok.... thats perfect.... I understand now. I will work on a script now and let you know by end of today

[seddik]

alright, good luck !!! for the script you mean for the wilayas.json and dairas.json ?!

[bensialih]

im doing the cities right now. I have pushed a branch that scrapes the osm_ids from openstreetmap. https://github.com/bensialih/algeria-cities/tree/feature/automation_script

I will push changes accordingly.

According to the findings of the script now running:- some of the entries cant be found some return city, village and county or a mix of those

We will have to choose what goes in there. Some will have to be manual inputs.

The script is now written and we can change it at a later date for the wilayas.

[seddik]

good, we can use this link to visualise missing cities. it works on the master branch only.

https://algeriacities.oecole.com

[seddik]

I can create a link the the feature new branch

[seddik]

You can use this to check things up on the automation_script branch, if it takes some time or gives a timeout error, then you need just to refresh, as it caches things and it may take long to retrieve data. when everything is cached it will take around 5 seconds to show the page.

https://algeriacities.oecole.com/feature/automation_script

to clear the geometry cache you can use these links for automation_script , master

https://algeriacities.oecole.com/clean/automation_script

https://algeriacities.oecole.com/clean/master

[bensialih]

@seddik Ive pushed my changes to the branch. getting an error. Can you have a look? https://algeriacities.oecole.com/feature/automation_script

https://github.com/bensialih/algeria-cities/blob/feature/automation_script/json/algeria_cities.json

[seddik]

@bensialih I have fixed the script, it was too much data, I am also on travel so my response would be a little bit slow.

I like your idea of the script, the map it is the esiest way possible, now we have to check which ones missing and fixing them.

[seddik]

I've added the wilaya number to check them wilaya by wilaya. let me know if we can add new features to it.

I'll share the script here in this repo later. it is made by laravel/php we can share it as a php code as well.

[bensialih]

this is great work on this. I have updated a lot of osm ids and had to do so manually. There remains about 120 missing osm ids. There is also the matter of some displaced or wrong osm_ids. Great work so far

[seddik]

alright I am back from my travels, i think it is better to separate the manual fixes by wilaya, and whoever want to fix a thing should assign the wilaya code issue to himself. I'll draw wilaya borders rectangle so when a city is worng it will show us that and null osm_ids as well, so fixing thing would be more precise

[seddik]

Alright I have added the rectangle for border to detect out of bounds cities, I've also showed the list of empty cities on the right, I've added a tool tip when you hover the mouse on the border of the city to know which city the cursor is selecting. I suppose these things should make fixing the cities a much easier.

so lets please use the issues cards of each wilaya and assign it to our selves before starting fixing things, so the work won't be done twice.

[bensialih]

this is excellent work @seddik hopefully I will finish the the cities today.

[seddik]

@bensialih I was afraid to do anything to not doing the same work on both your side and mine, so I waited for your response, If you want to handle all the cities no problem, I you want my help, you need to use the issues in the OSM miles stone.

[bensialih]

its ok.... let me finish them today and we can push this wheel forward. :-)

[bensialih]

@seddik I have finished all the osm updates manually. Some had differing spellings and that is why the openstreet api couldnt pick them up. Im guessing the next task is really to create a script to generate the GPS coordinates from the osm ids and put that into a file. I will start writing that script now unless you have it done already.

[seddik]

great work 🙏, no i didn't write the script for geometries, I think we should use issues to organize the tasks, we have to cleanup the current issues as well of the OSMID.

[bensialih]

script finished. I will run the script in 10 mins. It has a second gap between each request so i will let it run whilst I clean up the issues

[bensialih]

There is the small issue of the outliers. Cities with the wrong osm_id. the payload is a bit strange as well If you look at it, its a nested list inside a list inside another list. I will only extract the inner most list http://polygons.openstreetmap.fr/get_geojson.py?id=540555

[bensialih]

also.... do you want me to commit the scripts I used for your own use?

[seddik]

@bensialih yes that would be great !

[seddik]

@bensialih I can't find the correct algeria_cities.json with the fixes of (unfound cities and out of bounds coordinates).