Japan OSM improvement

[planemad]

Based on observations from #116. There are multiple data issues in japan which requires a comprehensive cleanup strategy with close involvement of the Japanese OSM community.

This ticket will track all the high level conversations and strategy on improving OSM data for Japan.

Existing data issues

• Most of Japan road network has been imported from Yahoo Japan in 2011
• The road data has positional errors (5-30m) when compared to Strava GPS and includes incorrect topology that does not match satellite imagery.
• Roads in metropolitan areas have been realigned to the correct position, but large parts of the country are still untouched since 2011.
• The Bing imagery coverage for Japan is comprehensive but does not match the OSM data or Strava GPS data. There is both an offset and orthorectification errors that varies throughout Japan.
• There is high resolution orthorectified imagery for Japan from GSI which perfectly matches Strava GPS data and is the best imagery source available. The coverage of this imagery is not complete.

Data improvement strategy

1. Select project area
   1. Priority: Urban areas > Highway corridors > Rural areas
   2. Check the accuracy of orthorectified imagery and Bing imagery by comparing with GPS data. Publish offsets if required.
   3. Evaluate extent of data offset and data freshness for the area
   4. Publish findings and communicate with local mappers to initiate a mapping task
2. Road realignment
   1. Create a road improvement task using the tasking manager
   2. Verify imagery alignment using GPS data
   3. For gridded networks, attempt to offset data to match imagery. If not successful delete imported unnamed roads and retrace area
   4. Delete imported roads with no match to imagery
   5. Realign all other roads and mark as finished
3. Road tracing
   1. Go through realigned tasks and invalidate to commence road tracing
   2. Trace missing roads and tracks from imagery and mark as finished
4. Validation

Next steps

• [x] Get feedback from Japan OSM community on above process
• [x] @planemad to set up trial map improvement task: http://tasks.openstreetmap.in/project/38
• [x] @planemad and @Rub21 to run task with team and closely monitor issues

cc @mapbox/team-data

[planemad]

Realignment trial report 09/16

• Today @pratikyadav @ruthmaben @nikhilprabhakar and @ramyaragupathy realigned 15,194 nodes in the Mutsu area
• Initial pass was to realign all roads in the task area, but due to inconsistencies in the reference sources and lack of GSI satellite imagery coverage, we could not verify that our work was accurate.
• Our revised strategy was to realign only highways (motorway-secondary) based on Strava GPS which was more accurate, but this created inconsistencies wrt to the existing data
• In the interest of keeping the improvement process simple for the community to contribute to, the conclusion was to sacrifice small positional accuracies to maintain large scale data consistency.

Revised data improvement strategy

1. Focus on high priority areas based on @Rub21 tweet map
2. Use the primary tracing source as GSI Japan Satellite (ort) where available and fallback on Bing in other areas. Most high priority areas have GSI Satellite coverage.
3. Realign road network only in urban areas to GSI Japan Satellite (ort), ignore offsets (<10m) wrt Bing
4. Trace missing roads based on tracing source

From tomorrow team-blr will try the above approach and report findings. A log of observations is maintained here

cc @mapbox/team-data

[lxbarth]

Wow, great progress.

In the interest of keeping the improvement process simple for the community to contribute to, the conclusion was to sacrifice small positional accuracies to maintain large scale data consistency.

@planemad - what does this concretely mean? Can you give an example for that?

[planemad]

@lxbarth to be as accurate as possible we were using Strava and OSM GPS to offset the Bing/GSI imagery for the best match, this is never perfect and results in an offset in another area. These continuous localized corrections which cannot be replicated exactly be someone else and adds to the data inconsistency since no source perfectly will match the data everywhere.

To make it simple, we're just going to use GSI Satellite or Bing as is without offset correction. We will skip realigning data with offsets under 10m from satellite imagery such as these and focus on adding missing roads.

[mapconcierge]

@planemad

I understand all steps. Yes, GSI ortho image layer (ort) has good accuracy.

電子国土基本図（オルソ画像）MaxZoomLv18 http://cyberjapandata.gsi.go.jp/xyz/ort/{z}/{x}/{y}.jpg source=GSImaps/ort

We have started special project team on Facebook Group with Asian addicted mappers, OSM teachers in Japan. My students will join. https://www.facebook.com/groups/697237443754399/

[mapconcierge]

@planemad

Mutsu city area doesn't have good GSI ort images and Strava GPS. We should choice

std 標準地図 MaxZoomLv18 http://cyberjapandata.gsi.go.jp/xyz/std/{z}/{x}/{y}.png source=GSImaps/std

[pratikyadav]

Realignment trial report 09/17

• Today @planemad @pratikyadav @ruthmaben @nikhilprabhakar and @ramyaragupathy realigned 6437 nodes in the Aomori region as suggested by @mapconcierge .
• The focus was only on merging/aligning highways (secondary+) to GSI Satellite imagery for pass 1.
• Tomorrow in pass 2 the team will focus on realigning the minor roads and compare the time taken with pass 1.

[unsungNovelty]

@mapconcierge, what kind of highway will the below road be? a service road, residential road or a living-street road?

https://www.openstreetmap.org/#map=17/40.785891873025996/140.7407283726662

[pratikyadav]

Realignment trial report 09/21

Today @planemad @pratikyadav @ruthmaben @nikhilprabhakar and @ramyaragupathy realigned 3201 nodes in the Aomori region.

The focus under pass 1A and 2 was -

• to create dual carriageways on major highways,
• realign and merge tertiary
• Adding missing tertiary+ highways
• Adding missing bridges/tunnels on tertiary+ highways;

Today under pass 3, the team will realign, merge and add missing unclassified, residential roads, living streets.

[planemad]

Realignment Summary

• Team of @pratikyadav @ruthmaben @nikhilprabhakar and @ramyaragupathy conducted a multi phase data correction exercise in Aomori
• We synced daily to break the data correction in multiple steps to prioritize effort to the most critical data first to least important and improve the process from our observations
• General observations:
  • Time taken to realign an object is proportional to the number of ways and nodes
  • By merging consecutive ways and removing unnecessary nodes in straight sections, the effort can be reduced
  • In areas with dense road network, it is faster to delete roads without extra properties (name, ref) and retrace than realign
  • Focusing a task on a limited set of features allows the use of filters to remove distracting objects
  • Time per user per task should be limited to under 60 mins, else there is a feeling of stagnation
  • Retagging roads should not be done without valid reason
  • At all points the larger objective should be to simplify the contribution of any newbie to OSM in Japan
  • Tasks must be selected away from active ones to avoid conflicts

Realignment proposal for Japan Based on these finding this is the proposal for a large scale road network correction for Japan:

• Select a project area with GSI imagery coverage and create a mapping task
• Notify OSM-JP of the task
• Make multiple passes of each area and make improvements in steps:
  • Pass 1: Realign and merge major highways (motorway-secondary); Create dual carriageways and link roads with correct oneways
  • Pass 2: Realign and merge tertiary; Adding missing tertiary+ highways and link roads; Adding missing bridges/tunnels on tertiary+ highways;
  • Pass 3: Realign, merge and add missing unclassified, residential roads, living streets
  • Pass 4: Realign, merge and add missing service roads, tracks and paths
  • Pass 5: Correct natural features polygons and waterways, add missing bridges, tunnels and viaducts
• The @mapbox/team-data will focus on completion upto pass 2 throughout Japan and support the community for further improvements

cc @mapconcierge @lxbarth @mapbox/team-data

[planemad]

Since we have Aomori completed till pass 2, the @mapbox/team-data is going to follow the similar procedure in the more dense Beppu-Oita region on Kyushu island from today http://tasks.openstreetmap.in/project/44

@mapconcierge would be great to have the OSM-JP community help with the validation of our work and feedback on the process.

[planemad]

While realigning highway junctions, make sure to fix any kinks on branching roads.

Fix any incorrect highway junctions where there is no crossing

.

[unsungNovelty]

@mapbox/team-data - Also merge and align the 4 way junctions which are split like this:

[samely]

We already finished http://tasks.openstreetmap.in/project/31#, which is a pilot task for realignment and adding roads in Chiba.

Those are the number of editions per user

User	Num OSM Objects	Num Changeset	Num nodes V1	Num nodes Vx	Num ways V1	Num ways Vx	Num relation V1	Num relation Vx
samely	77,862	333	45,220	18,416	5,856	8,370	0	0
RichRico	65,217	441	31,035	28,070	2,511	3,601	0	0
karitotp	64,290	215	32,538	21,848	4,531	5,370	0	3
Luis36995	53,561	429	24,981	21,864	2,751	3,965	0	0
calfarome	40,985	309	19,079	16,561	2,624	2,721	0	0
dannykath	38,549	653	20,531	11,481	3,563	2,973	0	1
ediyes	32,781	540	15,149	12,291	2,467	2,869	0	5
andygol	10,087	24	2,007	7,382	240	455	2	1
abel801	31,879	289	9,912	19,519	638	1,810	0	0
Rub21	1,815	31	759	711	131	214	0	0
pratikyadav	986	26	693	76	158	59	0	0
ruthmaben	767	18	236	405	58	68	0	0
jinalfoflia	311	7	240	5	52	14	0	0
nikhilprabhakar	310	5	254	9	34	13	0	0
srividya_c	294	17	167	14	75	38	0	0
ramyaragupathy	266	13	158	29	45	34	0	0
Chetan_Gowda	265	7	154	94	5	12	0	0
Total	420,225	3,357	203,113	158,775	25,739	32,586	2	10

[calfarome]

Team progress in Chiba Prefecture, Japan.

[unsungNovelty]

@mapbox/team-data : I have come across a lot of duplicate roads in the current task that we are doing (Ishikawa). For now we will be ignoring these duplicate roads and will be taking care of it in the later passes.

[jinalfoflia]

@mapconcierge, there is this tunnel like construction, what should it be tagged as?

Here is the link to the road.

[planemad]

Japan realignment progress map

https://gist.github.com/planemad/64da69b20ea9705b5673

• Stroke width indicates importance based on population/road density
• Colors indicate data status
  • Red: Not started
  • Yellow: Active
  • Brown: Pass 1
  • Orange: Pass 2
  • White: Pass 3
  • Green: Complete

[friedbunny]

@jinalfoflia Coincidentally, I've been through that tunnel many times. It's an enclosed "snow shed"/"snow shelter" (スノーシェッド or スノーシェルター) situated at a mountain pass with hills rising on either side.

Here are some pictures of this structure.

It seems like the OSM tagging is covered=yes.

[jothirnadh]

@mapbox/team-data : Whenever there is a pre-defined tunnel or a bridge such as one in the below image, we are not making any changes due to lack of evidence if the crossing is a bridge or tunnel.

[planemad]

Progress

• Task: http://tasks.openstreetmap.in/project/45
• Task area: ~8400km2
• Pass 1 completion: 3 days with 18 members (Mapbox)
• Pass 2: in progress
• Notes

cc @mapbox/team-data

[jinalfoflia]

@friedbunny Thank you :+1:

[jinalfoflia]

As there are multiple roads for re-aligning, I tend to miss a few of them. To better keep track of the roads that I modified, I use this style that highlights freshness of data.

• Before I started editing, we can see that the roads are not highlighted.

• After aligning them, I uploaded the data. After that I downloaded the same area again, this time the edits made by the me were highlighted and thus it'll be easy to spot the roads that were missed.

/cc @mapbox/team-data

[planemad]

@jinalfoflia nice tip and its always a good practice to upload/download data to keep it in sync.

[Rub21]

@mapbox/team-data : We've finished up the pass 2 in http://tasks.openstreetmap.in/project/45, let's start in new task http://tasks.openstreetmap.in/project/48

[mapconcierge]

＞ @Rub21 OK!! I've confirmed Fukushima Area. Our students team will join today. :-)

＞ @Rub21 Please add admin permission for Project 48 to me(MAPconcierge), I can translate in Japanese.

[Rub21]

@mapconcierge , That is great :) , could you give me your osm users, to add in Task Manager, hmm but I think first you have to login http://tasks.openstreetmap.in/, after it I can give you permission to access.

[lxbarth]

[mapconcierge]

＞ @Rub MAPconcierge http://www.openstreetmap.org/user/MAPconcierge

＞ @lxbarth That's great!! :-)

[mapconcierge]

Japanese Students are doing now!!

[planemad]

This is amazing @mapconcierge !

Has there been any confusion from the students, maybe you can translate the instructions in Japanese for the task?

[mapconcierge]

@planemad Yes, of course!! :-)

[planemad]

@mapconcierge sorry, thought you were already admin, done now. Also lets have subsequent tasks on http://tasks.teachosm.org with multilingual instructions, it would be great to have the community select the area that needs to be worked on based on imagery coverage map https://gist.github.com/planemad/64da69b20ea9705b5673 .

You can count on the full force of @mapbox/team-data to join in as soon as a task is up :rocket:

[abel801]

We don't have GSI imagery in Japan road improvement: Fukushima I think that we have finished the task.

@planemad

[Rub21]

@mapbox/team-data , I've added other task to improve Japan, #49 - Japan road improvement: Mito if you can not find more blocks to work in #48 - Japan road improvement: Fukushima can you jump to this new one

[mapconcierge]

＞ @planemad Okay thanks, I'm doing translation in Japanese. :-) http://tasks.openstreetmap.in/project/48

[planemad]

Mapping report of Ishikawa task

• Task area: 8000 km2 (density: 200-300 persons/km2)
• Pass 1: 3 days with 18 members (Mapbox)
• Pass 2: 2 days with 19 members (Mapbox)
• Notes

Key observations

• Multi pass approach worked well to focus effort required in the task to the most important features
• Confusion on how to handle blocks without satellite coverage: some were unlocked, some were marked as done. To avoid confusion, such blocks were validated to indicate they don't need work. An extra option to flag such blocks in a different color is required.
• Time wasted to mass invalidate blocks before starting pass 2. This operation took around 30-45mins for 8 members. An admin option in the task for such operations will eliminate this.
• Doing pass 2 separately helped identify incomplete work from pass 1. This helps in improving mapping quality.
• Objects modified in pass 1 are of higher priority than objects in pass2, but the effort for each pass is almost equal.

[planemad]

@mapbox/team-data based on the above observations and a voice discussion, it was decided to continue mapping in separate passes for future tasks. The most important reason for this is to detect quality issues from the previous pass.

[unsungNovelty]

@mapbox/team-data : Do not combine roads with different maxspeed tags

[pratikyadav]

Seeing a lot of holes/missing strips in GSI japan satellite imagery which are not actually one when we zoom in.

@mapbox/team-data

[pratikyadav]

Mapping report

Pass 1 and 2 done on two tasks for the parts with Japan GSI Satellite Imagery coverage.

48 - Japan road improvement: Fukushima

49 - Japan road improvement: Mito

Currently @mapbox/team-data is working on 54 - Japan road improvement: Osaka

cc @mapconcierge

[maning]

For later consideration, it would interesting to log who is editing in the TM outside the data team. This is just to highlight that this work is with the rest of community in Japan.

cc @mapconcierge

[planemad]

Important: Relations on ways combine without warning JOSM allows combining ways with different relations without any warning. In this example, roads with the same tags but part of different bus route relations were merged with no warning dialog. This will break the bus route relation. Do not merge ways that are part of different relations.

Bug report has been filed in JOSM trac: https://josm.openstreetmap.de/ticket/11945

cc @mapbox/team-data

[Andygol]

In order not to miss anything - use Todo Plugin

http://wiki.openstreetmap.org/wiki/JOSM/Plugins/TODO_list

1. Select required features
2. Add them into Todo List
3. Select and mark it one by one
4. Save result
5. Clear the list

[planemad]

Update

• Today we started on Kyushu Island of around 22,000km2
• Estimated completion: Pass 1: 9 days, Pass 2: 4.5 days

[mapconcierge]

＞ @planemad @lxbarth @maning @Rub21

I've translated @PlaneMad's post on diary of OSM.org in Japanese. http://www.openstreetmap.org/user/MAPconcierge/diary/36106

[pratikyadav]

For selecting continous way use Shift + W. It grabs the continous active way upto the next intersection.

Fliter out bridges ,tunnels and minor roads.

Just keep an eye on tags while merging.

[lxbarth]

I've translated @PlaneMad's post on diary of OSM.org in Japanese. http://www.openstreetmap.org/user/MAPconcierge/diary/36106

So amazing. Thank you @mapconcierge !

[mapconcierge]

Your too :-) ＞ @lxbarth Thanks for great challenge for Asian mappers.

[planemad]

@mapbox/team-data to speed up segment merging, JOSM can be configured to discard the yh:WIDTH tag automatically when editing. To do this search for the tags.discardable property in advanced JOSM settings and add yh:WIDTH to the list.

JOSM and iD will be updated in the next release to do this automatically for new users.

[planemad]

Update

• Pass 1 of Kyushu complete: 19 members, 5 days
• Pass 2 estimate: 4.5 days (Thursday)
• Next: Tokyo-Nagoya-Osaka mega region