Find missing routes in OSM

[HarelM]

We should implement a service that can do the following:

1. Given a GPS trace the server should be able to return a list of routes that needs to be added to the map.
2. after adding data to the routes (like type, color etc) the server should be able to commit this change to OSM database using the user credentials. This is a spin-off of #258.

GPS simplification article, this is the easy part: http://www.gpsvisualizer.com/tutorials/track_filters.html

I'm still looking for an algorithm that can find missing edges in a linked graph. I'll post things I though about. Using overpass might help with getting the existing data ,or maybe the OSM API itself.

[HarelM]

Current algorithm before our phone conversation:

1. Split route to 1000 points (issue here: splitting differently changes results)
2. Remove points that are close to an existing highway (this way the total data is reduced a lot).
3. Split remaining lines to lines without self loops (issue here: with loops within loops).
4. Add split lines only if they are not close to a previous line.

[HarelM]

After the phone call we though about adjusting the following:

1. Split route to 1000 points.
2. Remove points that are close to an existing highway
3. Piece back the remaining lines? Remove duplicate points here between remaining lines?
4. Use a different approach to splitting - on every point look in the previous point to see if there is a loop, if so, split the current line at the current point and continue with the remaining line.
5. Add split lines only if they are not close to a previous line. Interesting cases:

[HarelM]

@zstadler feel free to test out the improved algorithm (note that finial line merges are still missing). Latest algorithm:

1. Split route to 1000 points.
2. Remove points that are close to an existing highway
3. Piece back the remaining lines if the original split was due to 1000 point limitation
4. On every point look in the previous points to see if there is a loop, if so, split the current line at the current point and continue with the remaining line - ensures there are no loops within loops, and every line is loop-less.
5. Add split lines only if they are not close to a previous line.
6. Run simplification on all resulting lines: 6.1 Use D-P with low tolerance, 6.2 Modified version of Radial Distance - to remove points that are close to each other and have a sharp angle between their lines to keep "natural measured curves" and remove "GPS glitches". 6.3 merge simplified lines if possible (two close lines endings that are not near an exiting line).

[HarelM]

Added some basic naive merge after simplification. it improve the results but the merge should be a bit smarter... The end results are starting to look really good!

[HarelM]

I think I managed to nail the merge (I hope). Please try your best to make it fail, I dare you :-)

[zstadler]

The results do look much better. In particular, the backward search for self-intersection seems to work well.

The examples in the comments below are taken from the "Golchan" example, unless written otherwise.

I have found an unexplained gap:

Since this section was traveled from West to North, I'd expect the vertical segments to be kept connected during a backward search.

We also need to decide how to handle Y junctions with a returning branch, such as:

I prefer each branch to be a separate segment. Perhaps this should be handled as part connecting the segments to existing ways and between themselves.

Here is another challenge for that phase:

I've also noticed several tracks can have large gaps between GPS points which are probably issues with the GPS reception. It seems we have not considered them.For example, in the above Y junction example, there is a significant gap in the returning branch:

In another trace, the entire "new section" is a result of bad GPS reception:

I suggest that the backward loop search will also split segments when the distance between two GPS points is twice the "proximity distance".

Please reduce the tolerance for the D-P simplification much further. Here are the GPS trace (blue) the current simplification (red) and the Potlatch's simplification (yellow): Potlatch's results represent the track so much better, while the current simplification is no better than hand-tracing a satellite image.

[HarelM]

I have found an unexplained gap:

The gap is due to self loop and the algorithm stitched it (Y shape), I think it is a valid result, even if not an expected one.

We also need to decide how to handle Y junctions with a returning branch

Current implementation is to merge one of the branches to the continuing road. Otherwise you'll have too many parts, I think both examples are fine, I don't want to split them into smaller pieces.

there is a significant gap in the returning branch

If this is the same recording I treat it as a continuous line, I see no reason to treat it otherwise without causing issues elsewhere.

In another trace, the entire "new section" is a result of bad GPS reception

I don't think there's much we can do here, the user should remove this part.

I suggest that the backward loop search will also split segments when the distance between two GPS points is twice the "proximity distance"

I don't think it will be wise, we'll might get tons of small lines...

Please reduce the tolerance for the D-P simplification much further

It is on 5m tolerance - a value related to the accuracy of the GPS measurement, reducing it further will be as good as uploading the GPS itself to OSM, which IMHO is just bad practice (assuming most users are like me and are editing the OSM manually - this is just unfriendly). I think that the current simplification is in the middle between too much and too few points, I don't think we should keep all the points and reducing D-P's tolerance will do just that. Ariel tracing is enough for me - especially since it's very hard to notice the difference in zooms that are lower than 16.

I'm sorry I can't agree with you more, the given test case has almost every kind of possible edge case and I think the algorithm is handling it very well, If you have another case that you are less satisfied about please send it over, I think the current results for Golchan are the best you'll get.

[HarelM]

I was able to create a way today from test code (!) which is a very needed step if we are to automate the adding process to OSM. The code itself is fairly simple and works well, I'll continue by writing the code that manipulates the data in order to prepare it for adding and then add it. I'm working on the OSM dev server so no need to worry about adding unwanted stuff :-). I'm hoping to have a working version that can add ways to OSM by the end of the week. In order to reduce complexity, I think the easiest solution will be to add the new way using the closest node on the closest highway - assuming it is close enough.

[HarelM]

Initial glimpse on the add missing parts to OSM, Thoughts are welcomed:

[zstadler]

Simple and clean interface!

What about tracks for cars (tracktype=grade2 - שבילי קק"ל)? paved roads? Also, black and purple...

[HarelM]

I had the feeling some colors are missing... Is purple in use? I haven't seen it around...

I don't think paved and grade2 should be there, it will confuse the user, I think those 4 options are enough.

[zstadler]

Both orange and purple are for regional trails. Recently, purple was used for new trails at the Negev Highlands.

[zstadler]

Closed by mistake...

[HarelM]

Sure, no problem, I've added it too. Some more screenshots: Zooming in to one of the stars - it even tries to know what type of track it is by movement speed, I'm surprised it worked :-) I've moved to working against the dev server in order not to add things by mistake, I think it will be good if you open up a user there as well and see if you can run the site locally and test it there - I don't think we should test adding the trails on the production server just yet...

[HarelM]

Dev server doesn't have most of the DB the regular server ha, not sure I want to start importing data to it. I have published an initial version to test site.

[zstadler]

Please add a "source=GPS" tag to the added highways.

Please add a "created_by=IsraelHiking.osm.org.il" tag to the changeset tags.

The word "route" in the changeset description "Missing route finder algorithm" is inaccurate. I suggest using

Add a missing way from a GPS trace using IsraelHiking.osm.org.il

If you feel generous, it can be elaborated as "track" or "red cycleway" as in:

Add a missing [<colour tag> ]<highway tag> from a GPS trace using IsraelHiking.osm.org.il

P.S., My first changeset

[HarelM]

Reduced simplification tolerance to 3m, I think it's a bit too sensitive, but we can configure it later on if we want... Added the above detailed comment, I think it's a nice touch. Create by was updated and not added (I think this is what you meant to say). Added source=GPS to all added ways. So I think the only thing that is still open which I'm still not convinced how critical it is, is the connection between newly added way and existing one that is not by finding the closest point on an existing way but create a new point that is will create a 90 degrees angle between the old and the new way.

[HarelM]

Phone call summary: Current implementation adds highways when adding data to OSM, this can misguide the user since he can't be sure what's added to OSM. In order to overcome this issue the implementation should change in the following manor: Before presenting the user with the broken GPX data the algorithm should try and add small sections to existing ways (preferably if possible using points from trace), this sections will have a point that exists in another segment and should be linked when adding the data to OSM. Assuming the current algorithm creates cascading "L"s only an end of a L can be joined to another middle of L.

[HarelM]

Well the algorithm is more tricky, it doesn't really creates cascading Ls. However, I changed it a bit and now it does a better job, the only question I now have open will be described using the algorithm current pseudo code:

1. Go over a trace and find all the missing parts (this part splits the full trace to smaller pieces, removes points and piece is back).
2. Split missing parts when they have a self loop
3. remove duplication (since the loop are broken some part are now duplicate so we need to clear them)
4. [new] connect lines using the original trace (first extend current missing parts and then merge them if they have a joint point)
5. Simplify

The issue is with simplification since the connection lines use a point in another line which can be removed by simplification.

Possible solution which might change the results is to simplify before adding the connection lines but we did say that simplification should be last... Moving the simplification to the beginning might help reduce badly broken lines due to GPS position issues.

@zstadler ideas are welcome, other than that the algorithm is looking very promising.

[HarelM]

Algorithm was updated to main site. My initial tests are generating good results. @zstadler @valleyofdawn please let me know what you think.

[HarelM]

Added a few corrections to second simplification to adjust the line edges. I think the results are just amazing!

[HarelM]

Closing this long discussion revolving an amazing feature :-). All the things that I know should've been implemented are done, I'll open other issues in case of bugs.