Both proposal save files are way too brittle

[dabreegster]

A/B Street has map edits and the LTN tool has its own format for boundaries and filters. Both files currently reference OSM way and node IDs to refer to road segments. This is slightly better than the very original format, which used opaque RoadIDs that could change when import code slightly changed. The A/B Street map edit format even has backwards compatibility for the JSON schema itself, which is also non-trivial to maintain.

But the current format breaks constantly when downloading fresh OSM data, or with osm2streets transformations that merge or split roads, or when importing new footways. It's a serious headache to manually fix these up everytime, and if the LTN tool had lots of users, I'd be constantly breaking their save files.

So opening this issue to think about how to do better. Also very related to ATE schema (so CC @Robinlovelace).

An initial idea

How can we refer to a road segment over time in a stable way? I think the only hope is to use geometry instead. To reference a road segment, we use the polyline of its center, not any OSM IDs at all. That means when we load it, we have to snap, and the result could be multiple road segments. We have to decide how to handle edits to lanes -- if the save file's "old" lanes don't match the new basemap, several choices how to resolve.

This gets much more complicated for LTN boundaries / partitioning. I want to rethink how all of that works anyway (#916). Maybe a boundary is just a polygon, we let users draw these freehand, and we have a darn good snapping process that can trace roads appropriately and find the interior. Maybe the individual block addition/deletion process can still be an option to refine, or maybe not.

[dabreegster]

See https://wiki.openstreetmap.org/wiki/Permanent_ID for ideas

[dabreegster]

Also https://sharedstreets.io/how-the-sharedstreets-referencing-system-works/

[BudgieInWA]

I would love to see the SharedStreets reference system or something like it implemented at or around the osm2streets layer. As a project that is focused on multiple representations of roads, it feels appropriate that it should offer an opinion on how to refer to them across those representations.

After a cursory look, SharedStreets looks like the right level of abstraction -- hopefully it is fit for purpose. If it turns out to be good, it might be a good way to decide which OSM ways to combine into a single street.

As an end goal, it seems like a useful way for an end user to interact with osm2streets data -- such as defining LTN boundaries and modal filter locations.

[Robinlovelace]

Thought on this: the only reliable way is using spatial joins. Thinking about the seemingly common example where a way is split in 2 to add a new node/junctions, the 'old' way is still there in OSM, it just has different IDs. Using a buffer around the original feature can be used to create a 2 way lookup between what is in the local version and the updated OSM data. A decision needs to be made on the local data side whether to keep the single ID or to split it, as the version in OSM has. Matching all OSM ids that are completely within (GEOS function contains_completely() or similar) a buffer with a given radius around an original feature allows identification of the new IDs reliably. Allowing the user to specify buffer width allows some flexibility depending on how strict the requirements of geometries remaining unchanged are. In many cases the best thing to do will be to delete the old feature in the local copy and replace it with the new elements that exist in place of it. In cases where there are no matches completely within a given buffer distance of the old feature something else can happen (at some point the conclusion may be the way has been removed). Not sure if that idea is new or even if it makes sense, I tried to sketch it with limited success, happy to discuss: https://excalidraw.com/#room=5855d1e18e63ddc2c807,PSYriOaUY6_6VAJdVhSBwA

I had a look at the SharedStreets proposal a while back and never fully understood. Key test: try it and does it work? Same test applies to the idea sketched above, no guarantee it will work...

[Robinlovelace]

Having just looked at the OSM wiki on permanent IDs I think my idea is basically the same as https://wiki.openstreetmap.org/wiki/Permanent_ID#OSM_Permanent_ID_with_geo_URI but with a buffer around the 'geo URI' and including anything completely contained within that buffer, with a buffer width set depending on needs (e.g. 10 m, 5 m or 1 m for roads depending on the accuracy of local editing) rather than exact coordinates.

[dabreegster]

For a concrete example of how the save files work today, see https://github.com/a-b-street/abstreet/blob/master/data/system/proposals/stay%20healthy%20lake%20wash%20blvd.json. The road is identified as (OSM way ID, node 1, node2), the old lanes (left to right) are specified, and then the new ones. I'd love to get modal filters into this save file too. And all of this should eventually get "lifted up" into the osm2streets layer, as a way of describing a diff to a StreetNetwork.

I need to look into Shared Streets much more, but happy to consider it for a more robust way to identify road segments, especially if it's gotten wider traction as a standard.

Robin, I think your buffering idea is pretty close to what I have in mind ("snapping" or "map matching"). I can draw some sketches later, but imagine I have an edit describing a change to road A covering 5 city blocks originally. Someone updates OSM and splits that road into two pieces. When we snap, we can apply the edit to both pieces and detect which ones they are via map matching. Figuring out which roads an edit applies to is only one part of the problem.

Another question is how to resolve conflicts if the lane tagging changes. The save file format currently captures the "old" and "new" lanes for each modified road. The "old" is used to detect conflicts when the OSM data is updated. If a user changes a 4-lane road somehow, but later OSM is clarified to say that it's only 3 lanes, what's the appropriate way to resolve this? I'm not sure there's a one-size-fits-all answer. Maybe sometimes we can resolve this automatically, but depending on the application (in A/B Street or in ATIP), we might just need to ask the user what they want to do.

[Robinlovelace]

I have an edit describing a change to road A covering 5 city blocks originally. Someone updates OSM and splits that road into two pieces. When we snap, we can apply the edit to both pieces and detect which ones they are via map matching. Figuring out which roads an edit applies to is only one part of the problem.

Good to have a concrete example and yeah, which roads to edit only part of. There's also the attributes, aka tagging. I agree there's no one size fits all. One continuum I see is the sanctity of the original import vs the desire to keep up-to-date with OSM. In many cases I imagine the latter, wanting to keep up-to-date with OSM, is key. Notifications that edits may be lost, in the same way you get a notification on git that pulling down upstream changes will clobber local uncommitted changes, is a good solution, leading to the idea of an interactive merge conflict resolution.

I'm thinking of a UI that gives you options including:

• [ ] Update geometry to include changes in OSM
• [ ] Keep original geometry

And then if there is a clash in tag values, e.g. if cycleway=lane added by the user changed to cycleway=track in OSM on one of the newly split geometries, there could be a conflict resolution UI that says something like:

• Conflict in merging geometries, diff shown below
• [ ] Clobber local changes and synchronize with OSM (you can make local edits after to reflect local changes)
• [ ] Keep local attributes and ignore updates to the cycleway tag

[Robinlovelace]

Then it could go to the next conflict. Of course there could be settings like --clobber-all or --keep-local that could be used when there are too many changes to do manually but I suspect for many city locations updated every few months or so the changes will be doable via manual inspection. Thoughts?

[dabreegster]

Another type of problem to solve: right now the LTN file has all modal filters. I just added a missing one to OSM upstream, but all prior save files will erase it once loaded. Save files need to instead represent diffs -- add some filter here, or remove some filter that exists.

[dabreegster]

Another use case: @xzhou99 wants to import scenarios for infrastructure changes from GMNS. Current thinking is that the input would be a GeoJSON file with line-strings representing road center-lines, and properties on each one describing the lane configuration. (There's no need to agree on exact road centers -- as long as the geometry matches up well enough, it'd be OK.)

Besides describing simple changes like speed limit, the main thing to express is probably lane configuration. There are maybe a few different ways of doing this.

1) Overwrite the road's lanes entirely

Something like "from left-to-right, the lanes are [1.5m sidewalk, 2m cycle lane forwards, 4m driving lane forwards, 4m driving lane backwards...]"

2) Describe a diff from the road's current configuration

Like "Remove the parking lane on the left side and replace it with..." This could be a more attractive option, because the base data about current condition may differ across datasets, or get updated in OSM over time.

[dabreegster]

https://valhalla.github.io/valhalla/meili/algorithms/ explanation of map matching

[dabreegster]

https://gis.stackexchange.com/questions/204361/find-a-linestring-closest-to-a-given-linestring Hausdorff dist between linestrings may be useful

[dabreegster]

http://kuanbutts.com/2020/08/25/simplified-map-matching/