[UK] Reviewing osmlint-osmium and osmlint data issues

[sonyahilchik]

Background As part of on-going work to improve the quality of OpenStreetMap data, we recently used osmlint-osmium and osmlint for detecting issues related to road network data in UK. In total, we are going to review issues in 9 categories.

Our plan is to review all detected issues and fix it in OpenStreetMap. There's a link to the OSM Wiki page.

Timeframe We are going to start review on the end of August - September 2021.

Tools We will use existing OpenStreetMap editing tools, we will use #hashtag for the project #mbx_linters_UK

Editor: iD or JOSM

Satellite Imagery: Bing, Maxar, Esri

Street Level Imagery: Mapillary, OpenStreetCam, Other open sources if available

Contact Person We welcome feedback, suggestions, and insights from local mappers. If you have questions about this work, or a specific edit by our team, please reach out to Natallia (natallia.zhouner@mapbox.com) or post a comment in a changeset.

Linter Output for UK

Issue	Total	Data
Crossing Highways	854	crossing_highways.zip
Island Highways	1157	island_highways.zip
Impossible Angle	350	impossible_angle.zip
Impossible Oneways	414	impossible_oneways.zip
Mixed layer	54	mixed_layer.zip
Missing Type Restriction	57	tr-missing_type.zip
Missing Role	1046	tr-missing_role.zip
Excessive Roles	261	tr-excessive_role.zip
Invalid Role	116	tr-invalid_role.zip

@Zetkun @abrohood @shvrm @lukasmartinelli @vladaboitsik

[systemed]

Do you have a list of the categories you're working in?

Bear in mind that access implications on footpaths/cycleways are very UK-specific and it is easy to break this data (as Amazon Logistics have done, repeatedly). If you are changing highway types you might want to ask the local community for advice. The UK community tends to use the talk-gb@ mailing list and not forum.openstreetmap.org.

[sonyahilchik]

Hi @systemed !

There's a list of the categories: crossinghighways impossibleangle impossibleoneways islandshighways mixedlayer tr-excessiverolestr (turn-restriction) tr-invalidroletr (turn-restriction) tr-missingroletr (turn-restriction) tr-missingtyperestrictiontr (turn-restriction)

Thank you for notification, we will keep in mind words about footpaths/cycleways and changing highway types.

[SomeoneElseOSM]

There's a list of the categories...

@sonyahilchik If that was supposed to communicate what you are planning to do, it didn't help. Just to take one example from that list, can you explain (on talk-gb please) how your software will detect something that is "mixedlayer" and why it is a problem?

[sonyahilchik]

Hi @SomeoneElseOSM ! It is detected by different conditions which describe logical mistakes in mapping. Example: mixed layer category. It describes features where a highway crosses another highway with a different layer tag. There're roads with different layer values. It's obvious there that these roads are on the same layer (=0) and we need to fix it.

This info also duplicated in talk-gb.

[SomeoneElseOSM]

There're roads with different layer values. It's obvious there that these roads are on the same layer (=0) and we need to fix it.

Well it can't just be about different layer values, can it? It's very common (and entirely correct) for an implicit layer 0 road to become a layer=1 road over a bridge and then back down to implicit layer 0 again. Presumably what you're actually detecting here is joined, crossing ways (and if so, it's really important to know what sort of ways are considered for this check)?

Also, how do you know what the fix is? In the example above, without context, it's not actually clear what you changed and why. Please link to the relevant OSM object rather than just including a cropped browser screenshot.

This info also duplicated in talk-gb.

Presumably you're expecting it to show up at https://lists.openstreetmap.org/pipermail/talk-gb/2021-August/thread.html#27498 ? If so, it hasn't appeared yet.

[sonyahilchik]

@SomeoneElseOSM message will appear in 'mapping activity' thread started by me. This problem is about crossing ways that have different layers tag and, for instance, intersection point (what is wrong). Here's changeset of this example: https://www.openstreetmap.org/changeset/110460358

[SomeoneElseOSM]

(crossposted from talk-gb https://lists.openstreetmap.org/pipermail/talk-gb/2021-September/027773.html )

Hello Sonya,

In restrictions sometimes there can be missing roles, and we delete it, because it's broken

No, as has already been said over at https://lists.openstreetmap.org/pipermail/talk-it/2021-September/072960.html that is not the correct approach.

You have been given lots of advice on mailing lists and Github issues, which you seem to have mostly ignored. Your mappers also seem to be ignoring OpenStreetMap changeset discussions (to take just one example - https://resultmaps.neis-one.org/osm-discussion-comments?uid=10398649 has no replies to the 8 questions asked in the last fortnight).

This behaviour is not OK. Please ensure that all your employees immediately reply to the questions that other mappers have asked them, and that you follow all the advice that you have been given. Please also ensure that your mappers review all their previous changes made in this series of organised edits and ensure that they repair any damage that they have caused (some, like https://www.openstreetmap.org/changeset/111251617 have already been repaired by local mappers).

Failing to do this would mean that we (the DWG) will be forced to prevent further edits by those editors damaging the database in this way until they have been suitably trained. We may also need to revert some of your teams' recent edits. In the meantime we will alert those mappers we can identify who have not replied to comments asking that they do so.

(snip)

Best Regards,

Andy (local GB mapper, and also a member of OSM's Data Working Group)

[sonyahilchik]

Hi @SomeoneElseOSM ! We have already seen this message in maining list and we will answer to it soon. Thank you for your crosspost.

[matkoniecz]

Note that

(1) taking validator reports (2) making low quality edits that hide validator warning and damage OSM data (rather than fixing actual problem) (3) refusing to undo or fix edits after receiving info that edit damaged OSM data (4) commenting

Hi! thank you for your feedback. Correct me if i wrong, please. Best regards!

is insulting, not acceptable, not helpful and utterly pointless.

What kind of instructions were given to employees? Are they paid/measured based on how many errors detected by validator were silenced? Note that "validator was silenced" does not mean "problem was fixed".

Are you planning to undo damage to data that you caused and was spotted by local community actually caring about map quality? Rather than expecting to mappers that you disturbed will clean up that damage?

And why service road entrance that clearly leads to a garage or underground parking was mapped as entrance=yes?

This specific is about https://www.openstreetmap.org/changeset/110279678 but it seems that many other edits have similar problems

[vladaboitsik]

Hi, @matkoniecz ! We addressed these issues. Also will prepare more specific documentation about how we work with the Linters and about our approach in resolving issues mentioned in feedback. Always open for your comments and feedback.

[boothym]

This problem is about crossing ways that have different layers tag and, for instance, intersection point (what is wrong). Here's changeset of this example: https://www.openstreetmap.org/changeset/110460358

In this example you've reduced the length of the road with layer=-1 so that it no longer goes beneath the building with layer =0. https://overpass-api.de/achavi/?changeset=110460358

Surely the better approach would be to split the road at the building and leave the layer=-1 on the bit of road which might be below a building? As you can't be sure from the aerial imagery that there isn't a road beneath the building.

And why service road entrance that clearly leads to a garage or underground parking was mapped as entrance=yes?

Especially when the name of the service road is literally "access underground car park"!

[matkoniecz]

Always open for your comments and feedback.

What about other feedback on edits of your employees made before 2021-09-01? Is it completed or still ongoing?

[sonyahilchik]

Hi, @boothym. Thanks for mentioning, it's our mistake, we changed the edit according your notes. New changeset is here. In the case with 'entrance=yes' tag, we already fixed this and answered on changeset.

Hi, @matkoniecz. For now we don't have unanswered comments for our changesets. We set up internal statistics of answers and our employees check their e-mail every day. Our reply can take 1-2 days because of big differences in time zones or weekend/sick leaves/vacation a bit more.

Thanks for your feedback, we are always open for your comments.

[sonyahilchik]

As we wrote above, our team prepared some documents about how we work with the Linters and about our approach in resolving issues mentioned in feedback. Here you can find more details about how we get and process linters, their types, examples, announcing our work to the communities and so on. Here is the links on these wiki pages: https://wiki.openstreetmap.org/wiki/Mapbox_linters_types_description#Intro https://wiki.openstreetmap.org/wiki/How_Mapbox_Data_RAVE_team_work_on_receiving_feedback_from_OSM_users

[sonyahilchik]

The team is going to start the 2nd iteration of improving the quality of OpenStreetMap data. Now we will use internal Mapbox data validation jobs that search OSM data for errors. We got 5113 detections of issues which we're going to review.

We will update this ticket with results of our work.

[SomeoneElseOSM]

Before you make any edits, can we please see those "5113 detections of issues"?

Thanks for posting to talk-GB as well, but if any of those issues also affect Northern Ireland please also mention it on talk-ie which tends to cover the whole island of Ireland, not just the Irish Republic.

[systemed]

Yes, please upload a GeoJSON file or similar somewhere so that the UK community can take a look.

[sonyahilchik]

@SomeoneElseOSM @systemed here are json files by categories, you can watch them through geojson.io Issue	Data
Crossing Highways	GB-crossinghighways.zip
Island Highways	GB-islandhighways.zip
Impossible Angle	GB-impossibleangles.zip
Impossible Oneways	GB-impossibleoneways.zip
Mixed layer	GB-mixedlayer.zip
Missing Type Restriction	GB-tr-missing_type_restriction_tr.zip
Missing Role	GB-tr-missing_role_tr.zip
Excessive Roles	GB-tr-excessive_role_tr.zip
Invalid Role	GB-tr-invalid_role_tr.zip

[SomeoneElseOSM]

you can watch them through geojson.io

Thanks - for the benefit of anyone else, that site seems not to accept pasted JSON but does work when a file is dropped on the browser.

However, can you please document how you have detected the content of each of the files above and how you plan to fix them? For example, how will you fix https://www.openstreetmap.org/relation/4167920 ?

What about https://www.openstreetmap.org/way/521496421 ?

[sonyahilchik]

@SomeoneElseOSM we use principles of osmlint and osmlint-osmium, you can read more about how we detect issues in this ticket above. Also on the wiki page and here in the comments above you can find links to description of our work.

We make fixes only if it's reasonable and have groundtruth source. In case of restriction you sent, we won't fix it because restriction is not valid, the road connects to the oneway and there's no groundtruth sources like Mapillary, etc. The second case also won't be fixed because of grounds lack for correction.

[SomeoneElseOSM]

@sonyahilchik Thanks - it makes sense to remotely edit neither of those, I think (although the latter one I will review on the ground as a temporary traffic scheme has just been made permanent in that area).

[SomeoneElseOSM]

Interestingly, GB-crossinghighways.geojson contained an entry for what is now https://www.openstreetmap.org/node/9439598425 (a misssing service/service junction) but not https://www.openstreetmap.org/node/9152958285 , which was a missing service/footway one.

[SomeoneElseOSM]

I wonder why an "impossible angle" has been detected at the end of https://www.openstreetmap.org/way/448233297 ?

[vladaboitsik]

Hi @SomeoneElseOSM ! The impossible angle was detected because a small segment of a road has a duplicate. You can see it if you drag the white node. I didn't fix it to let you check this case.