TobiPeterG / OSMTraffic

Overview
GNU General Public License v3.0
8 stars 2 forks source link

Mapping Real-World Data to OSM #4

Open mvglasow opened 4 days ago

mvglasow commented 4 days ago

From README:

One major challenge is ensuring that traffic data received from various sources (Datex2, APIs) accurately maps to road segments in OpenStreetMap (OSM). This includes handling mismatches in road names or coordinates.

The solution I built (TraFF, various sources, Navit for navigation) addresses that.

Referencing is primarily done through reference points. The TraFF format, which I developed for this purpose, allows only WGS84 coordinates (aka GPS coordinates) for reference.

Other attributes, such as highway type, ref, name etc., are optional and added mostly to make the locations human-readable (“S61 between Wysokie and Raczki” is easier to read than a set of coordinates). They can also be used as auxiliary data to match locations to the map when the point data is ambiguous.

Mapping a location pair to a set of OSM ways (or parts of ways) is done using essentially the same routing algorithm the navigation app already has, with a few tweaks:

For our sources, we would have to convert everything into WGS84 coordinates:

The beauty of using coordinates (possibly along with auxiliary identifiers) is that the “producing” and the “consuming” end may use different maps – whether that is maps generated from OSM data at different dates, or even maps from completely different sources.

TobiPeterG commented 4 days ago

Wow, thank you for the great write up :)

Have you had a look at the prototype server already? I have an example data source that provides json and the server converts it to GEOjson currently, but this is really just the first draft to have a starting point.

I would like to have the output format of the API something apps can understand and work with as soon as possible, so we can focus on adding data sources and ways to report data instead of rewriting this part. It seems like adopting TraFF and their format would be useful. :) Do you have a link to it?

Any help on that front would be appreciated. :)

TobiPeterG commented 4 days ago

Ah I found it, it's on gitlab, right?

That looks really good :) I saw there is a WIP server already, but with the last update from 3 years ago. Is there a reason the development stopped? Should it be used as a base instead?

mvglasow commented 4 days ago

Correct :-) The entry point is http://traffxml.org. The Resources section has a full specification, with an explanation of why and how. Hope it’s useful – I had a similar idea a couple of years ago (and implemented some things), but for the foreseeable future I won’t be able to invest a lot of time in this. Still, a lot of work already went into this and I’ll be happy if someone takes over.

Apps has links to the apps which work with the format.

Resources has some general information about TraFF. There are also some converters (in Java) for some traffic sources. That includes Datex2, as well as proprietary formats used by the road authorities for Poland, Lithuania and Latvia.

https://gitlab.com/traffxml/traff is the repo for the specification, as well as the event list.

https://gitlab.com/traffxml/roadeagle/-/wikis/Data-Sources has a list of potential data sources I once compiled. It may not be up to date.

As for stopping development on the server, the reason was lack of personal time. I may not be an expert on writing web apps, thus the approach I have taken may not be the best – see what makes sense and use it. Java was chosen mostly because I started with an Android app, for which I had already written some libraries for a number of sources. However, they are somewhat limited when it comes to resolving locations, especially those based on milestones. This could probably be done a lot better with access to a full OSM map (the libraries use a table of milestones with coordinates, and guesstimate the location of kilometric points in between).

TobiPeterG commented 4 days ago

Thank you for sharing this :)

I would love to have you on board, it seems like you are very knowledgeable in this area. This doesn't mean however that you would have to invest time if you don't want to. :)

I would like to take this project step by step. So first a data aggregator of available sources that converts the data in a format that apps understand. Then app developers could start to actively adapt it.

Afterwards, a way for apps to report their own data would make sense in my opinion.

And maybe also offer a custom routing service? But that doesn't have priority in my list, as the navigation apps could figure that out.

What do you think?

mvglasow commented 4 days ago

Sounds great. I’ll be happy to help, only constraint being time.

Step by step sounds good. As for a data format, I am not aware of anything widely understood by OSM-based (and FOSS) navigation apps, TraFF was the first attempt at this.

RoadEagle and Qz both are aggregators of some kind, albeit somewhat limited and in the format of an Android app, publishing traffic news in the form of Android broadcasts. So, if app devs want to adopt it, they can do so already. If the data format stays the same, switching from these apps to a HTTP-based service would just be a matter of switching the transport mode.

And yes, that was roughly my approach: start out with already available sources, build an aggregator for them, then provide a way for users to contribute data.

Especially for that scenario I think it is important to keep the traffic components (server, API, data format) independent of apps, in order to allow different apps to use it and thus widen the user base.

TobiPeterG commented 4 days ago

Yes, this sounds reasonable :) I'll have a look at RoadEagle and Qz, do they include the data aggregator in the apps or do they connect to a server for this?

What API design would you recommend?

mvglasow commented 4 days ago

Both are aggregator apps.

RoadEagle periodically polls various online services (each serving a different area and/or different types of events), converts the messages to TraFF format and adds them to its database. Expired messages get purged periodically. Apps can subscribe to updates for a certain area and receive a feed of all active messages for the area. When RoadEagle receives a feed that has new messages, it sends its subscribers a feed with the new or updated messages for their subscription area.

Qz connects to an external TMC receiver and receives traffic updates via RDS-TMC. The update mechanism is older – as messages are received one at a time, they are broadcast immediately, but I had plans of implementing the subscription mechanism à la RoadEagle there as well. RoadEagle is closer to what would be needed for a web API.

I have designed an API which supports multiple transport modes – HTTP as well as Android broadcasts (others, such as D-Bus, could be added if needed). There is a draft specification, which you can find in the TraFF spec repo (https://gitlab.com/traffxml/traff). The theory is pretty much complete; HTTP transport should be, too.

Basic features are:

Currently no “desired“ poll interval is specified. One could add two extra data fields, sent by the server when setting up the subscription:

In the spec, you’ll find some mentions of deprecated operations (mostly related to subscriptionless push on Android). This is what I originally did in Qz but wouldn’t lend itself well to an HTTP API.