Kerala Road mapping with ML data

muzirian-zz commented 6 years ago

In late July 2018, severe flooding affected Kerala state in India due to unusually high rainfall during the monsoon season, Kerala's worst floods in nearly a century over 373 people died within a fortnight, while at least 280,679 people were evacuated, rescue operations are going on. kerala floods wikipedia

Two most used maps in disaster response https://www.microid.in/keralaflood/ which is used to identify flooded roads and https://keralarescue.in/map/ , https://process.keralarescue.in/rescue used to visualise rescue requests are using OSM data. OSM coverage for kerala is far from complete and needs lot of work.

In response to the disaster mapping efforts Facebook have shared machine learning based road data for Kerala, and offered help of their mapping team to add this data to OSM. But we would need people with local knowledge working and validating the edits especially tagging.

This ticket is to discuss the collaborative effort with FB mapping team and the local community mappers. All discussions from other communication channels will be compiled here.

Road data - https://fb-public.box.com/s/ggo37k7bvj92jj137wr4hpmd5ba5rjt1 - Please download to evaluate the data. Do not import to OSM yet.

Next actions

[x] Post this ticket to community channels for discussion.
[ ] If consensus to import was made, archive all public Tasking Manager projects related to roads.
[ ] Create a new TM project to consolidate all road mapping work.
[ ] Community along with FB Maps team to import/conflate/validate the data (details of workflow, TBD)

muzirian-zz commented 6 years ago

Posted the ticket to forum and talk-in mailing list. Will be updating forum link as it goes live after moderation. osm wiki page for 2018 floods

maning commented 6 years ago

@planemad Are you doing a quick visual check? Can you post observations here? @muzirian The dataset covers the whole state of Kerala, can you identify the priority areas (geojson) so we can quickly prepare TM projects as soon as community gives a 👍 .

planemad commented 6 years ago

Checked 3 files randomly. Observations:

Definite overall increase in road coverage by atleast 50%, but far from achieving complete road coverage
New roads are mostly minor residential roads and tracks. Most connecting roads are already in OSM
The roads that are detected are almost accurate (high precision)
Many minor roads are still not detected as they are obscured by trees (low recall)

new1

Issues

Many unsurfaced roads and tracks are classified as unclassified instead of track or path

Birdges seem to be auto generated whenever a road crosses a waterway. This mostly works except when the waterway is badly traced and incorrect bridges are created.

A large portion of the residential road network is still undetected due to dense canopy through most of Kerala.

The undetected road causes frequent gaps in the detections that need to be manually connected.

geochetan commented 6 years ago

Gone through few output files both in urban and village areas. Some observations are:

We can rapidly import and fix minor alignment and connectivity issues in rural area.
Urban output is bit challenging due to complexity of existing data but takes the triple the effort of village areas.
90% of the road classified as unclassified and need to convert to either residential in case of urban area, and track or path or residential in rural area.
There are lot's floating roads which are not connected nowhere and small segments which need to be extended further.
As @planemad pointed out, this will definitely improve Kerala road network. What we need is dedicated import accounts to have clean import and good channel (maybe gitter) to get started with.

Few screenshots:

Urban area

OSM Existing Data

ML Output (Pink is ML data)

Check gif to know coverage in urban area https://media.giphy.com/media/8Z2khz7sOWw9mwrB4X/giphy.gif

Rural area

OSM Existing Data

ML Output

Floating roads - easy to fix

Missing connectivities and other errors

planemad commented 6 years ago

Based on ^ the dataset is a significant addition over what is present on OSM. The only errors introduced are ones due to existing data issues on OSM. Tracing roads in Kerala manually is super hard and the ML output looks very promising to ease the process.

With a little bit of manual work connecting the gaps and fixing validation errors in existing OSM data, this is worth importing 👍

naveenpf commented 6 years ago

Kerala PWD (https://103.35.198.65:8181/kpwd.gis/) has details of NH,SH and MDR. Officials told they will check if they can change the license. They wanted a proposal from our side describing how to it will beneficial for Govt. (during flood times and long term) Is there any similar proposals to Govt ?

naveenpf commented 6 years ago

NH - 1800 Km SH - 4300 Km MDR - 27,000 Km

DrishT commented 6 years ago

Thanks for the quick analysis. Spot on with all the findings, as we usually take care of all those mentioned fixes in our editing process.

maning commented 6 years ago

❤️ the eval @planemad @chtnha! Feels like data is good and we can start thinking about import process. @muzirian do you have the priority areas?

What we need is dedicated import accounts to have clean import and good channel (maybe gitter) to get started with.

@chtnha Is this critical at this stage? AFAIK, FB team has dedicated accounts to do such work. Community mappers shouldn't have to create a new one IMO as long as we have clear changeset metadata.

muzirian-zz commented 6 years ago

Thanks a lot for the analysis Arun and Chethan. The data looks promising.

@maning Today I checked the the rescue request map at https://process.keralarescue.in/rescue It is helpful but lots of people were trying to make requests for victicms, but forgot to change the geotags to correct location. And there are other high priority zones to be considered. Will go through it again and will check with other people to help with and try to mark something tonight.

muzirian-zz commented 6 years ago

Talked with Jaisen and Manoj to identify most affected places. River basins along Pampa, Chalakkudy and Periyar rivers are most affected with floods. Made a coarse geojson of those places.

Also districts of Wayanad [Relation: Wayanad (2018203)] and Idukki [Relation: Idukki (1819274)] was hit with floods along with landslides.Attaching district boundaries also.

Also referred bit.ly/ckdyflood

boundaries.zip modified_aoi.zip

muzirian-zz commented 6 years ago

Made a geojson by referring rescue requests on https://process.keralarescue.in/rescue Tried to avoid some wrongly geotagged request, might still need modifications. request.geojson.zip

naveenpf commented 6 years ago

It looks good. Let us start importing.

jaisuvyas commented 6 years ago

Data from FB looks promising.. Yes.. River basins along the Pampa, Chalakkudy and Periyar, as well as Idukki district and Wayanad district are the most flood- affected areas. These places and the areas around Chengannur town and Aluva town needs special attention in this regard.. Kuttanad area between Alappuzha and Kottayam districts was almost completely submerged and don't know how much destruction occured for the basic infrastructure there. Needs attention there too.

Akhilan commented 6 years ago

Radomely checked the data and looks promising for me. I think we can proceed.

ranjithsiji commented 6 years ago

Yes we can go. May be some problems will be fixed later

manojkmohan commented 6 years ago

Supports. We are also trying to mobilize field level peopls to contribute to make osm better. in the flood time we maitiained 2 maps for codinating local operations. Flood level mapping and others information in Chalakkudy River Basin bit.ly/ckdyflood. community pooled data of Relief camp locations in thrissur disticts bit.ly/thrissurflood. its used google map because of lack of data in osm. hope in future we can do better.

manuvarkey commented 6 years ago

I have checked out the dataset and the level of detail is good. Manual editing is required for certain features. We can go ahead.

muzirian-zz commented 6 years ago

The plan is to use the help of FB's mapping team to import using their internal tools that takes care of conflation, connectivity fixes, etc. So every edit will be manually reviewed and will have to pass validation checks by tools. They will be following the workflow described on https://wiki.openstreetmap.org/wiki/AI-Assisted_Road_Tracing With Indian road tagging scheme: https://wiki.openstreetmap.org/wiki/India:Tags/Highway

https://wiki.openstreetmap.org/wiki/Talk:AI-Assisted_Road_Tracing#Kerala_Mapping_-_August_2018 < We can use this part of wiki for discussion Facebook import process has already been reviewed and approved, Only difference will be the extra validation by community on tasking manager. To avoid conflicts, will be archiving current active roads projects and setup a new project on tasking manager.

If no other concerns we will be proceeding with this plan.

bgirardot commented 6 years ago

I am going to create the requested import specific wiki page if no one minds.

i think we can fill most of it out from the existing wiki pages liked to above, but the Imports email list asked for a specific wiki page for this import, i think we can do that pretty easily.

Edited to add: muzirian and I connected, we using the import page he started.

muzirian-zz commented 6 years ago

Will be using https://wiki.openstreetmap.org/wiki/Kerala_Road_Import for import documentation. cc @bgirardot

osm-in / mapping