Fix sidewalk tagging

[lxbarth]

Based on the sidewalk analysis happening over on #125 we should fix tagging them. Use osm-sidewalker identify and fix all highway=footway that are sidewalks or crosswalks which are not tagged in OpenStreetMap.

• Add footway=sidewalk to highway=footway that are sidewalks either separated by grass patch or attached to the associated highway.
• Add footway=crossing to highway=footway where footway crosses highways.

  Task Workflow

Updated mapping instructions are documented here

[lxbarth]

Let's be careful with adding a footway=sidewalk tag to a highway=footway that contains a sidewalk relation to a road - it may not be established mapping practice.

[tcql]

I've got a slippy map of New York, DC, California, and Germany using the current logic here

@planemad can you take a look at this and see if you can do anything with it? Let me know how you think we could improve our sidewalk detection, and if there's anything I could do to improve the map / how we share this information

[planemad]

@tcql looks great. Was digging through the map and have some observations:

• Motorways don't have sidewalks, so its pretty easy to drop these false positives.
• Most false positives seem to be in leisure areas like parks, stadiums and around lakes. These areas usually have detailed footways and the chance of a false positive is much higher, Maybe they should be dropped too?
• Other obvious false positives
• Footways that actually require sidewalk tag seem to come up in clusters, where a particular mapper might have done micro mapping.
• Multiple parallel footways is a sure sign that some of them need secondary tags

Summary

• Overall i'm seeing a lot of false positives and a majority of them are footways in leisure areas and open spaces. There are probably a few legitimate cases in there but looks like we can drop them
• Wether a footways is a sidewalk can sometimes be detected visually by seeing the arrangements of all footways in the vicinity. Some subtle coloring for other footways can help investigate these patterns further
• The most actionable areas are those with a large number of hits. These seems to be neighborhoods of micro mapping and allows us to make many more fixes in single sweep. In the initial phase I think this is where our focus should be rather than fixing stray sidewalks one by one.
• The workflow to do this task as a team is currently not very effective. For starters we need a single click way to open the map location into JOSM.

[tcql]

@planemad thanks! I'll hit as much of this this morning as I can

Motorways don't have sidewalks

:+1: removing motorways

Overall i'm seeing a lot of false positives and a majority of them are footways in leisure areas

That's tough to narrow in on - roads can cut through leisure areas, or run along the edges and have their sidewalks marked inside the park area

I'll see if there's some way to tighten this without excluding everything. If this turns out to be a big issue, we could just exclude footways in leisure areas from the search, and hit the obvious ones elsewhere for now.

Wether a footways is a sidewalk can sometimes be detected visually by seeing the arrangements of all footways in the vicinity. Some subtle coloring for other footways can help investigate these patterns further

Color all footways, subtly marking which are sidewalks, which are paths, etc?

The workflow to do this task as a team is currently not very effective. For starters we need a single click way to open the map location into JOSM.

I'll look into this today

[planemad]

I'll see if there's some way to tighten this without excluding everything. If this turns out to be a big issue, we could just exclude footways in leisure areas from the search, and hit the obvious ones elsewhere for now.

What could quickly help is highlighting parks and water areas on the map. That way its easier to investigate them visually on the map which is effective in taking a decision.

Color all footways, subtly marking which are sidewalks, which are paths, etc?

:+1: nothing too drastic. I feel the eye might be able to spot odd patterns more easily this way.

For starters we need a single click way to open the map location into JOSM.

Just remembered theres a great hack by @geohacker which could help: http://geohacker.in/unconnected-tiger/

It uses surface API to get the osm_id of the object clicked in a Mapbox dataset and loads it into JOSM using remote control. If you could quickly port it to gl-js that would be perfect.

[tcql]

@planemad

Build a little prototype here

• show parks
• show water
• exclude motorways from sidewalk checking
• supports click to view data, & a button to open in JOSM

I'm running a tile-reduce job for the whole US which should be complete later tonight, and then I'll update the map style

[planemad]

@tcql :+1: beautiful!

[tcql]

updated the map with the continental US

EDIT: it's important you load the page as http:// not https:// - It makes an ajax request to JOSM remote control over http. If you visit the map as https, the request to JOSM remote control will fail because of https -> http security blocking. If you want to use it via https, you might need to enable unsafe scripts. In chrome that looks like this:

[maning]

Per voice with @srividyacb and @jinalfoflia, we will run a couple of test cases today and Friday using @tcql's map. To easily monitor discussion, all encountered false positives will posted in #125. Discussion on the final workflow will be here.

[maning]

[WIP] Workflow for sidewalk tagging

@srividyacb and @jinalfoflia is currently testing @tcql's map to fix sidewalks and flag false-positives. For easy tracking, false positives will be reported at #125. This is to improve the detection of sidewalks.

However, on our initial pass today, we think that working on each detected way would be too onerous. The best way is to assign people to work on a larger area and fix it in one go. This allows the team member to look into the general context of the area and easily tag connected ways.

Sketch Workflow

1. @tcql will prepare another layer showing the density of errors at z15 or z16 grid, this can either be using kernel density or convex hull in turf.js. Add a JOSM RC button to download data. The detected errors are highlighted. select=object[,object...]
2. Each person will be assigned an area say DC grids.
3. In JOSM, download data within the assigned grid.
4. Add the sidewalk map as JOSM background: tms url tms:https://{switch:a,b,c}.tiles.mapbox.com/v4/tcql.usa-sidewalks/{zoom}/{x}/{y}.png?access_token=pk.eyJ1IjoidGNxbCIsImEiOiJaSlZ6X3JZIn0.mPwXgf3BvAR4dPuBB3ypfA
5. Add an inverse filter for all generic footways, ie. type:way highway=footway -(footway=sidewalk | footway=cycleway | footway=crossing)
6. Use JOSM's todo plugin to work on each highway=footway and add appropriate tags.
7. Upload and move on to the next grid. Changeset comment: #Sidewalk tagging. Project part of https://github.com/mapbox/mapping/issues/126

@tcql, will run a daily update so we can visualize what needs work by next day. We will test this workflow by Friday here in @mapbox/india-office cc @planemad @lxbarth @Rub21

[tcql]

• [x] Set up daily runs
• [ ] Hook daily runs into interactive map
• [ ] build density grid
• [ ] daily stats for detected errors (for monitoring progress, csv dump like date,errors for now, should be fine)

Got hung up on other things the past few days, tonight is my test run for the daily cron script, and tomorrow I should get the density grid worked out.

[maning]

@tcql, we tried adding your map as background image in JOSM, but it doesn't show your glowing green style.

[maning]

@lxbarth Before going full steam with the above workflow, should we first inform the talk-us community? Especially since sidewalk tagging is tricky and there is no consensus within the list.

[maning]

we first inform the talk-us community?

Done. Waiting for community discussion. https://lists.openstreetmap.org/pipermail/talk-us/2015-October/015590.html

[tcql]

@maning I moved the final location of the interactive map to OSM Sidewalker. This supports selecting z15 tiles and opening them with segments selected in JOSM, as well as opening individual segments in iD

todo still is to get daily updates flowing into the map (I found a few issues with the way I was pushing up the data), and then a daily stat counter for how many sidewalks we're detecting / fixing

[RicoElectrico]

Is it possible to get that map in an international version? (on a side note, it's funny how USA is assumed implicit at Mapbox and others. Reminding you that there is world beyond the States :stuck_out_tongue: - please consider this for all your other activities and development.)

[tcql]

@RicoElectrico definitely! It's on the list. We started in the US intentionally because some other areas tend to follow other types of tagging, and so we could have a good basis to start from

[maning]

@tcql Excellent! Tile download is really helpful. To recap our workflow:

1. Select a red tile in the map. Click, Open in JOSM.
2. Add an inverse filter for all generic footways, ie. type:way highway=footway -(footway=sidewalk | footway=cycleway | footway=crossing)
3. Add the selected sidewalks in JOSM's Todo list.
4. Review and add footway=sidewalk.
5. Click Mark and move to the next in the list.
6. Upload to OSM with
   • changeset comment: #Sidewalk tagging. Project part of https://github.com/mapbox/mapping/issues/126
   • source: Bing, Mapbox Satellite, osm-sidewalker

[maning]

2015-10-30

Today, we fixed 536 detected sidewalks:

• @srividyacb - Texas - http://overpass-turbo.eu/s/cmX
• @jinalfoflia - California http://overpass-turbo.eu/s/cn0
• @chtnha - New York http://overpass-turbo.eu/s/cn2

@tcql, can you give us an approximate number so we can benchmark how much time/effort is needed for the team to fix this?

Issues

• A lot of sidewalks need to be split-ted if it goes through a road intersection to add footway=crossing
• There are footways that has turn restrictions relations (example) is this common in the US? For now, we don't edit them.
• We ignore footways with tags like bicycle=yes, area=yes and named footways.
• OSM-sidewalker map bug - Download in JOSM pop-up does not appear when the map is rotated (not north-up).

@tcql @lxbarth @planemad

[srividyacb]

2015-11-02

Today, we came across lot of fixed sidewalk from Friday.

• @srividyacb - Texas - http://overpass-turbo.eu/s/cmX
• @jinalfoflia - California http://overpass-turbo.eu/s/cn0
• @chtnha - New York http://overpass-turbo.eu/s/cn2
• @maning - North Carolina http://overpass-turbo.eu/s/cq5

@tcql @lxbarth @planemad

[tcql]

@srividyacb thanks, I'm working out some issues with automatically updating the map. should have it sorted out today so we can see proper updates

[srividyacb]

@tcql In sidewalk detection we are getting some places where highway=footway is also having area=yes tag.

Also there are highway=footway with a tunnel=yes tag. These should not be highlighted to mark as footway=sidewalk because they don't run alongside of main highways.

Link to the area: https://www.openstreetmap.org/#map=15/29.75993/-95.36531

[tcql]

just FYI to @RicoElectrico and anybody else wondering, international sidewalk detection is up on the interactive page now.

Daily updates are still stalled at the moment (particularly because we're working on some tweaks to tile-reduce and OSM QA tiles). I'm occasionally running manual updates. Some tweaks to improve false-positive matching are going to go in today and we should have a fresh update by tomorrow.

[jinalfoflia]

@tcql we came across quite a few sidewalks with the bicycle=yes and name tags. In the talk-us mailing list, they have discussed that named footways are not sidewalks. It would be great if we could eliminate them while processing itself.

/cc @maning @srividyacb

[maning]

we came across quite a few sidewalks with the bicycle=yes and name tags

Here's the running list of secondary tags that should be excluded: https://github.com/mapbox/mapping/issues/125#issuecomment-157022204

[jinalfoflia]

Sidewalk tagging report

• Till date there are 1,37,875 ways which are tagged with footway=sidewalk and footway=crossing.
• @maning , @chtnha , @srividyacb and @jinalfoflia have been working on tagging missing sidewalks and crossings in the US and have tagged 3,146 ways.
  • @srividyacb - http://overpass-turbo.eu/s/cUP
  • @maning - http://overpass-turbo.eu/s/cUT
  • @chtnha - http://overpass-turbo.eu/s/cUU
  • @jinalfoflia - http://overpass-turbo.eu/s/cUV
• The team also was involved in recognising false positives in sidewalk detection and improving the detection process with @tcql

Next Action[s]

Our goal is to complete tagging of missing sidewalks in the US region by the end of the coming week.

• [ ] We want @mapbox/india-office to join hands in completing this task.
• [ ] Let's work on sidewalk-tagging this Thursday [26 Nov 2015].
• [ ] Tomorrow @srividyacb and myself will be giving an introduction on sidewalk tagging to the entire team.

/cc @planemad @maning @srividyacb @tcql @lxbarth @shvrm

[tcql]

@jinalfoflia thanks for writing this up! I'll make sure to push an update with the latest version of the map today to eliminate as many false positives as possible

[srividyacb]

As the osm-sidewalker is not working for the bounding box and team is finding it difficult to work, we have pinged @tcql on this and we will start the sidewalk tagging as soon as it gets fixed. (Ticket open in osm-sidewalker)

cc @planemad @lxbarth @tcql @maning

[srividyacb]

Updated Instructions

1. Each person will be assigned an area.
2. Click on highlighted area on the map and download the data to JOSM.
3. Add the sidewalk layer in JOSM to verify the sidewalks detected before tagging: tms url tms:https://{switch:a,b,c}.tiles.mapbox.com/v4/tcql.sidewalks-world/{zoom}/{x}/{y}.png?access_token=pk.eyJ1IjoidGNxbCIsImEiOiJaSlZ6X3JZIn0.mPwXgf3BvAR4dPuBB3ypfA
4. Use JOSM's todo plugin to work on each highway=footway and add appropriate tags.
5. Upload and move on to the next grid.

Changeset comment: Added missing footway=sidewalk tags to sidewalks. Project: https://github.com/mapbox/mapping/issues/126 Source: Bing/Mapbox

Filters to use

• Add an inverse filter for all generic footways, ie. type:way highway=footway -(footway=sidewalk | footway=cycleway | footway=crossing)
• highway=footway - name=* : Highlights all the footways
• footway=crossing : Hides all the footway=crossing tagged footways once added.
• footway=sidewalk : Hides all the footway=sidewalk tagged footways once added.
• You can switch between filters to hide/show the ways and add tags correctly and change the incorrect tags.

Do's and Dont's

• Let's be careful with adding a footway=sidewalk tag to a highway=footway that contains a sidewalk relation to a road - it may not be established mapping practice.
• Don't add a sidewalk tag to the footways which have names to it ( Filter: highway=footway | name=*)
• If a footway is having a bicycle=yes tag, we will be adding the footway=sidewalk tag if it's a sidewalk as per the imagery. Don't remove bicycle=yes tags
• We won't add sidewalk tag to the highway=cycleway
• Don't align or extend the existing footways.
• The sidewalk detection algorithm has already removed false positives like tunnels, cycleways, highway=footway with area=yes and other possible tags in the process. If you come across tags like that ignore them.

cc @planemad @lxbarth @maning cc @mapbox/india-office

[tcql]

Quick update on the tile-reduce side: I'll be working to get an automated solution using scheduled Autoscaling, so we can run sidewalk analysis daily. Stats will be posted here

cc @maning @srividyacb

[lxbarth]

@tcql fyi, only US needed for now in case this is much faster to set up for US than for entire world.

[nickidlugash]

Per chat with @lxbarth, it looks like DC could use a bit of clean-up, as we're seeing some small fragments of footpaths that should maybe be tagged as sidewalks. E.g.:

http://www.openstreetmap.org/edit?pk_campaign=mapbox-edit#map=21/38.88771/-77.01634

and

http://www.openstreetmap.org/edit?pk_campaign=mapbox-edit#map=19/38.89729/-77.01196

/cc @srividyacb @maning @chtnha @jinalfoflia

[planemad]

@tcql we don't have a good way to measure our progress on the sidewalks:

• We currently manually assign areas to team members, but this is not scalable in the long run or to coordinate tasks with the community. If blocks that have been edited are highlighted in a different color it would help skip areas that have been looked into previously.
• The analysis includes a significant portion of false positives. This makes it hard to estimate at what point the task is complete. If there is a way to rank the detected sidewalks with some score, it will help focus our efforts much better to fix the easiest ones faster and on a larger scale.

cc @shvrm @srividyacb @maning

[planemad]

Proposed sidewalk rendering on openstreetmap-carto

[tcql]

We currently manually assign areas to team members, but this is not scalable in the long run or to coordinate tasks with the community. If blocks that have been edited are highlighted in a different color it would help skip areas that have been looked into previously

@planemad It might be possible to use a dataset to store features (selected tiles), like the Chennai Flood Map and then load that dataset on the frontend. I'll look into it.

The analysis includes a significant portion of false positives. This makes it hard to estimate at what point the task is complete. If there is a way to rank the detected sidewalks with some score, it will help focus our efforts much better to fix the easiest ones faster and on a larger scale.

Yea, I get that this is difficult. We could possibly use a level of variation in the distance from the road along a segment to score it's likelihood to be a sidewalk. I ticketed this on osm-sidewalker

[planemad]

It might be possible to use a dataset to store features (selected tiles), like the Chennai Flood Map and then load that dataset on the frontend. I'll look into it

@tcql this would be great. The use case to know if someone has looked into a tile is pretty common for similar mapping analysis and tile reduce based tasks that we can run in the future and is well worth the effort.

[srividyacb]

@mapbox/india-office

Here is the top cities which we need to hit for sidewalks. Assign a city from this list of top Strava activities in US. Work in pair if its a big city and mark as done in the table.

City	Assignee	Status
DC	@oini	:white_check_mark:
New York	@srividyacb @nikhilprabhakar	:white_check_mark:
SF Bay	@jinalfoflia	:white_check_mark:
Seattle	@ramyaragupathy	:white_check_mark:
Madison	@abhisheksaikia and @jothirnadh	:white_check_mark:
Detroit	@jothirnadh	:white_check_mark:
Chicago	@ruthmaben	:white_check_mark:
Texas	@srividyacb	:white_check_mark:
Denver	@ramyaragupathy and @maning	:white_check_mark:
Boston	@jinalfoflia and @nikhilprabhakar	:white_check_mark:
Los Angeles	@oini	:white_check_mark:

[nickidlugash]

So excited to see this continued work on sidewalks! I'm happy to help do some reviews after clean up, especially for the cities I've lived in (DC, Boston, Philadelphia).

[planemad]

@nickidlugash are you doing the review using the Strava style? It would be nice to have a style specifically to do QA and help visually spot any glaring tagging issues.

Tried this out with this style to differentiate footway, sidewalks and crossings:

This helped spot a bunch of incorrect tagging, like these sidewalk tagged as a crossing.

[ramyaragupathy]

I have been working on sidewalk tagging in Denver city with this workflow. This helps me set a reference and a sense of where the work is happening within a city.

[srividyacb]

OSM Diary post up: https://www.openstreetmap.org/user/srividya_c/diary/37594

No actions here. Closing

[tcql]

thanks everybody who ran on this :+1: this is some seriously awesome work. It's been awesome watching the stats on this and seeing the massive progress everybody made.

I would love to revisit the tooling in the future and implement some of the ideas here to improve this. I also believe we can definitely improve matching and remove more false positives so the community can run with this and keep making OSM better

[RobJN]

Are you marking false positives so they are hidden from the slippy map? Or is it just a case of remembering where you have already looked?