track changes for streets

[kpwebb]

Per conversation with @migurski et al here's our draft proposal for "track changes":

As street networks change, the SharedStreets referencing system updates ShStRefs for altered or newly created streets, and generates metadata describing the changes. Users subscribe to change event metadata allowing internal street-linked data to stay in sync with current street definitions.

Change events include:

• Addition of newly created, or previously unmapped street segments;
• Deletion of streets that have been removed (or were incorrectly mapped)
• Alteration of streets, including changes in geometry/length (due to improved mapping), or reclassification.
• Splitting of existing street references into multiple segments due to the creation of a new intersecting street segment.

Changes in OpenStreetMap are tracked continuously based on OSM changeset events. Changes can also be submitted from registered SharedStreets users that maintain proprietary or government street basemaps. Users can submit new SSRs and metadata without sharing underlying map information.

[schnerd]

This appears to make a lot of sense at a conceptual level, and definitely something that would be useful to keep historical street-linked data up to date. Questions:

• Are combine events supported? Or are these just treated as deletions + addition?
• How would a user "subscribe" in practice (webhook endpoint, json/xml feed, etc)? Or is this still being debated?

[kpwebb]

@schnerd , I'll follow up in #23 about ID stability, but just to address the questions above:

1) Let's add combine. We got to this list because splitting is common/likely (building a new road or adding a missing road to a map results in a split). Deletion + combine is less common but happens (map was wrong and is getting fixed, or road is removed).

2) Open to suggestions on good pub/sub protocols and practices. Most of the techniques I'm familiar with are geared toward consumer API streams (e.g. webhooks, push notifications, etc.). The only enterprise-oriented practices I'm familiar with are tied to AWS infrastructure (SNS/Kinesis).

We're looking at SNS internally and already use Kinesis, but not sure if that's a solution for non-AWS users. This seems important: what is required to support global scale subscriptions? Could we support that over something that's built on an open protocol bound to AWS/GCE/similar?

[emb]

Let me know if am off base here. I am pre-supposing a data store that has SSRs with metadata.

Once an a reference is generated it should never be deleted. This is to ensure backwards compatibility within the system. This is not to say that we should not mark a reference as no longer valid (new additions, removal or more accurate reference elsewhere). Invalid references could potentially point to the superseding valid ones.

With regards to sharing modifications something like Kineses which has up to 168 hours of retenntion period maybe good for live updates, however, we should consider a feed of some sort that will allow us to bootstrap an existing dataset from scratch.

[migurski]

+1 to immutable ShSt LRs, geometries, and intersections.

I am also a big fan of the OSM Planet approach, with a static filesystem and multi-grained replication streams linked from https://planet.openstreetmap.org/replication/.

[migurski]

I wrote up a napkin sketch of Addition and Deletion on a very simple street network and noted a few questions that came up: https://gist.github.com/migurski/c44ba8f0914c1fa8d04e39e3c1b6a0ee .

• The formats for add and delete operations are like text file patches.
• Should changes cascade to related entities?
• Can we derive these patch formats by comparing two states?
• Can we chain operations to reduce them into smaller operations?

More in the link. Small illustrations included, though Github doesn’t want to display SVG in Markdown; scroll to the bottom to see them.