Integrate Traffic Data from different sources (e.g. from the German bundesAPI)

[jwsp1]

There is an API for traffic data for the German highways: https://github.com/bundesAPI/autobahn-api (in German).

I guess it is much effort to implement the data but it can still be a first step of an enhancement to know about heavy traffic or closed highways.

Upvote & Fund

• We're using Polar.sh so you can upvote and help fund this issue.
• We receive the funding once the issue is completed & confirmed by you.
• Thank you in advance for helping prioritize & fund our backlog.

[dumblob]

This (and other countries) is an essential necessity for real world car navigation. It is also the only reason I stay with Google Maps.

Related topic is rerouting based on traffic jams computed from the real-time position of users of OM (of course with asking them first to provide this data voluntarily) and the expected speed on the given part of the track (taking into account all traffic lights, turns, etc.).

[hermann-san]

OSM and has the same issue of missing traffic data. Maybe there can be some collaboration with them or some hint's on how to make some progress https://github.com/osmandapp/OsmAnd/issues/6878

[SlouchyButton]

I'd like to add that for Czechia it is possible to retrieve the traffic information for free in XML (and DATEX II, but never heard of that) live - https://dopravniinfo.cz/cs/pages/data-from-jsdi/traffic-information. They should also contain (apart from traffic jams) all traffic incidents (crashes, closures etc.) and expected time to finish the event (finishing works on road or removing crashed cars).

I'd propose that for a sake of usability, the navigation algorithm should be changed, so it can work with traffic information and create some way in which we could implement individual traffic provider sources into the app. My knowledge with C++ and C is very limited for any in depth help with this, but I'd very gladly try to implement the Czech traffic information.

Down the road the system could be expanded with optional sending of current location with speed to servers for "in-house" traffic data not depended on any 3rd party.

[biodranik]

Of course, the server should collect and pre-process traffic data, not millions of clients. Otherwise, that free public API will be DDOSed and closed.

It is a good idea to make an aggregate issue with a list of all publicly available sources/countries, so we can have a good starting point.

Are there already some aggregators for information like that?

[SlouchyButton]

@biodranik I don't think there is in this project. When researching I found one source of these traffic APIs for different countries, but it seemed outdated (was on OSM wiki if I remember correctly). I propose making a wiki page with a table where we could link all the public (and/or private) traffic APIs for future reference. I could try and look up if other countries (except DE and CZ) has something like this and make more comprehensive list.

Personally, I'd try to gather information directly from the official sources of individual countries. I expect most of them should be free, since it is official source. Going the route of paid plug-in, or subscription service doesn't seem a viable option, because there are many alternative free navigation apps. Having this pretty crucial feature behind paywall would inevitably hurt the app.

You mentioned a server collecting this data and processing it. If needed, I can provide server for backend as I have business in web dev, and I own a physical server located in the Czech Republic, Brno. (Have no idea what are current options for backend servers for this app)

Regarding the OsmAnd and OrganicMaps collaboration (as proposed by @hermann-san ), there is one possibly mutually beneficial way of collaborating. If both apps implemented the optional location and speed tracking for their users, both apps would have much bigger user base together. This would result in much more accurate information about traffic, than would be possible to achieve only relying on the user base of one of the apps. I would understand any privacy concerns regarding this feature, but this could be implemented in a privacy respecting way and on an optional basis. Together with open source nature of app, it could be fully transparent and work mainly as a supplement for the official sources.

[baarkerlounger]

In England National Highways provides a similar API that can be used free of charge for traffic data in DATEX II format.

https://www.trafficengland.com/services-info

They also publish Java and C# libraries for processing it https://github.com/SaturnEclipseUK

[dumblob]

Hey everybody!

I just stumbled upon an open source project which seems to actively gather real-time traffic data in a multinational manner.

https://traffxml.gitlab.io/ (https://gitlab.com/traffxml )

Thery also have an Android app RoadEagle.

Could you get in contact with them ( info@traffxml.org ).

Btw. Navit seems to use their traffic data. @metalstrolch @bkoppelmann @jkoan @OLFDB @mvglasow and others could maybe know something more.

[mvglasow]

Yes, TraFF is my project (so far I’ve been alone on this, but contributions will certainly be welcome).

Just to give you an idea of what TraFF is: at its core, there is an open data format for traffic information, designed for fully machine-parseable event information, independent of the underlying map.

On the software side, there is a library to parse and generate feeds in the TraFF XML format, as well as import libraries for a bunch of formats, including the Datex II format, which is widely used in the EU.

As for the software ecosystem, so far there are three workable products:

• Navit, a navigation system which processes TraFF data (this feature is currently available on Android only)
• RoadEagle, an Android app which reads traffic information from a number of sources and makes them available on screen as well as in TraFF format (currently with support for Poland, Lithuania and Latvia)
• Qz, another Android app which can receive TMC data via an external hardware device and makes it available on screen as well as in TraFF format (works in a number of places in western an d central Europe)

A TraFF server (which would aggregate data from various sources and make them available in TraFF format via an API) is in the works. Future clients could then directly connect to a TraFF server, rather than tap into sources locally. This would also allow for integration of a bunch of additional sources, which are not feasible to process with an Android app (e.g. the Datex-II library relies on components not available on Android, and the Czech service mentioned above requires running a server at a static IP address).

The source code for all of this can be found at https://gitlab.com/traffxml (the only exception being Qz, which lives at https://gitlab.com/mvglasow/qz.)

Main obstacle to implementing all of this is my own limited free time – as I said, if anyone would like to participate in the further development of TraFF, I’d be more than happy.

As for the German API, I have come across that before. While its authors look official (“Bundesstelle für Open Data”), they are actually an activist organization. I am not sure where exactly they pull their data from and under what terms. It might be a legal gray area in the sense that they do not have a formal license to use the data, but since all it was ever intended for is to power a free-of-charge app, nobody has objected to this usage so far.

However, Germany does offer some sources of traffic data which are officially free to use. Visit https://mcloud.de and search for “Datex”, which will give you a bunch of sources in Datex-II format. (The TraFF Datex-II converter will convert that into TraFF, though you might need some external data to decode the locations.)

As for TraFF, there is a list of potential sources at https://gitlab.com/traffxml/roadeagle/-/wikis/Data-Sources.

[dumblob]

Potentially related: https://github.com/organicmaps/organicmaps/discussions/4102 (Overture Maps Foundation - seems governed by Linux Foundation).

[gerion0]

As for the German API, I have come across that before. While its authors look official (“Bundesstelle für Open Data”), they are actually an activist organization. I am not sure where exactly they pull their data from and under what terms. It might be a legal gray area in the sense that they do not have a formal license to use the data, but since all it was ever intended for is to power a free-of-charge app, nobody has objected to this usage so far.

The "Bundesstelle für Open Data" only documents the API for data that are provided officially by the government. In this case let me translate the README of the link Autobahn App API:

The BMVI (official department) built a completely useless app. An autobahn information app See: https://autobahn.api.bund.dev/

Really completely useless? No, since there is a public API for up-to-date data with regard to construction side information, webcams, parking places, ... - surely useful for someone.

So to say, while the API documentation is inofficial, the information itself comes directly from autobahn.de which is a official website of the Autobahn GmbH which belongs to the government.

[xephos1one]

This might be useful: https://github.com/graphhopper/open-traffic-collection

[dikkechill]

The above mentioned TraFF project (https://gitlab.com/traffxml) provides quite a number of building blocks already:

1. Open data format specification (TraFF XML) to normalize/standardize various data feeds to
2. Mappings from common public data feed formats (e.g. Datex II) to TraFF XML format
3. A server implementation for collection and pre-processing public data (TraFF server) with proof of concepts for Belgium, Germany, Finland, Latvia, Lithuania, Luxembourg, the Netherlands and Poland

For the longer term this still requires:

• TraFF XML implementation in OrganicMaps
• Server(s) running for OrganicMaps (and maintenance of the infrastructure)
• Support on TraFF project (e.g. adding and updating data sources, ) - seems currently a one person who has done a lot of work

Maybe it's possible to build a PoC with the current traff-server (or a sample traffxml file) and a traffic layer (view only) in OrganicMaps?

[biodranik]

Thanks, that's an interesting project.

1. What are pros/cons of TraFF vs other implementations (if there are any)?
2. Why only one person is working on it? Who else supports it?
3. What are server requirements?
4. How server handles millions of clients and a large number of requests all over the world in busy peak hours? How does it scale? Are there any load tests/performance results?
5. How many users are already using it? Can we share a server with many other project to get a better-quality data?
6. Who will own this data?

[mvglasow]

Happy to hear you’re interested in my project. Regarding your questions:

Pros/cons

Pros:

• Fully machine parseable and language neutral events: Events are expressed through predefined codes rather than freeform text – unlike Bing and Waze which rely on freeform text for most of the details. This makes them easy to parse and easy to present in any language – it is just a matter pf providing a lookup table of predefined strings to display a TraFF message in any language.
• High level of information detail: A message (i.e. a report about a situation affecting one particular stretch of road) can be expressed by any combination of some 200 event codes, some 200 qualifiers as well as quantifiers for numeric values. Events would tell you that there is a roadworks area, with a reduced number of lanes and a temporary speed limit; quantifiers would tell you how many lanes are closed and what the speed limit is. Again, Bing and Waze do so only in very rudimentary form, and much of the important information is only available as freeform text.
• Easy-to-parse location and direction information: A location is primarily expressed through 2–3 points in WGS84 coordinates, as well as directionality information (one or both directions). Other attributes, such as road class, road number/name and geographical area, can be added but are only meant as auxiliary information. WGS84 coordinates are a must, as they are presumably well understood by any GPS-based navigation system. In comparison to other locations:
  • TMC requires recipients to obtain a location table in order to decode locations. While this saves bandwidth, it makes the information useless without the location table and limits the locations which can be encoded.
  • Some sources (e.g. autobahn.de) do not designate both message endpoints: either one endpoint is missing, or it is not clear which direction the message refers to.
  • Other sources (e.g. Datex-II) offer encoders a lot of flexibility to use their preferred location scheme (options including TMC-style location tables or coordinate pairs in the encoder’s preferred coordinate system), but that puts a massive burden on the processing end.
• Built on top of a well-defined data format – XML is one of a handful of standard formats. The schema is intentionally kept simple – in contrast to, say, Datex-II.
• Conversion libraries are available for a number of other formats (Datex-II as well as some custom ones; TMC has also been converted, albeit the code is not in library form at the moment).

Cons: Adopting TraFF is going to be pioneering work. The project is still in an early stage, not last because it is currently one guy (myself) doing this in his spare time. However, any help will be very much appreciated and I will be happy to support adoption into a widely used project.

Community

As mentioned above, so far no one has come forward with major contributions. The ecosystem so far consists of:

• Navit, a navigation app which has support for TraFF data (on Android, it can receive data from Qz and RoadEagle)
• Qz,. an Android app which connects to a Bluetooth TMC receiver, retrieves messages, converts them to TraFF and feeds them to apps
• RoadEagle, an Android app which retrieves traffic information for (currently) Poland, Lithuania and Latvia (which use custom formats) and feeds them to apps

It appears to me that most FOSS-minded people don’t own or drive cars, and car drivers tend to go for what’s available – which is mostly proprietary at this moment.

This is why I have been reaching out to navigation apps, as I expect they have the user base with the biggest interest in that kind of data, while also being in a position to contribute.

Server

Word of caution: the server is still in early development stage (the conversion libraries and Android apps are more mature), and no load tests have been performed yet. Requirements are:

• Java servlet engine (tested with Tomcat)
• Database backend (development is currently on HSQLDB, with plans to move to PostgreSQL)
• Linux OS preferred, others are untested

This also means there is no live server yet. If OrganicMaps can help out here, that would be great. Ultimately, sharing a server between multiple projects is encouraged as expenses would be shared while the benefits would be multiplied.

On the other hand, the feed format and conversion libraries do not depend on the server. Also, the TraFF specification includes a local delivery mechanism for Android – one app connects to a non-TraFF backend, converts messages to TraFF and announces them via a broadcast, which any app with support for it can pick up.

Data ownership

The data sources I am using so far are under various open data licenses, some with and some without a share-alike clause. I would like the data to be available in a spirit similar to OSM itself. That primarily means the information should be free to use and redistribute. Share-alike is desirable as long as it doesn’t limit our access to data sources.

[biodranik]

@mvglasow Thanks for the detailed answer! More questions:

1. How suitable is this approach to collecting data from drivers all over the world, and providing both: a) live information that helps to build a faster route right now, and b) statistical data from the past for better route planning? How hard would it be to implement it? Are there any existing projects/source codes for these tasks?
2. Why Java for the server?

[mvglasow]

Crowdsourcing data was the original idea and TraFF was developed with that in mind. Tapping into other sources was more of a quick win so one could start out with a usable set of traffic information at least in some regions (and even with crowdsourced data, I envision the two kinds of sources might very well complement each other).

As of TraFF 0.8, the mechanism for users to report traffic data back to a central repository is not implemented, simply because I haven’t gotten around to doing it, but this is definitely planned for future versions.

One point to address in this case is authentication: the current “downstream-only” version of TraFF does not require any permanent user ID. When users wish to receive updates, they subscribe to a certain set of areas and road categories and receive a randomly generated subscription ID, which is valid only for this particular subscription (typically for one trip, but ultimately controlled by the subscriber side). The first feed contains all currently active messages matching the selected criteria, after which only “delta” feeds are sent to save bandwidth. As subscription areas identify where the user currently is and/or intends to travel, associating them with a natural person should be prevented. By using a random ID which changes with every new subscription, data exposed to the server is essentially just the IP address, timestamp and subscription area (all of which are pretty much unavoidable). Also, the ID cannot be used to compile a ”history” of a certain person’s travels.

With user-supplied messages, this may change as we may need a way to keep out the bad guys – anything from sloppiness to outright malicious insertion of bogus data. At the bare minimum, we can block a known bad user (or silently ignore their messages) if we’re alerted to some form of abuse. We might also consider some form of reputation: a message which is confirmed by other sources increases that user’s reputation; if a message is at odds with what other sources report, it decreases it. That way, when correlating messages from multiple users, we can give more weight to messages from trustworthy users.

At the same time, we still want to minimize privacy impact. I suggest that even for authenticated users, subscriptions should not be associated with a user ID (only user-submitted messages should), and any association between a user-supplied message and the user ID should be discarded when the situation to which the message refers has been resolved.

Another point will be correlating data from multiple sources but referring to the same stretch of road. Ideally, the server should detect when multiple messages affect the same stretch of road, make sense of the information and process this into one single message. For example:

• An official source reports roadworks on a certain stretch of road and a driver in that same section reports a tailback, with the speed at which traffic is flowing and the coordinates where the tailback ends.
• One driver reports a traffic jam they got into, with an accurate speed at which traffic is flowing and the end of the tailback when they encountered it. Another driver on the opposite carriageway (where traffic is flowing freely) sees the obstruction and reports where the “tip” of the queue is. Together, these two messages will give us the length of the tailback, along with a speed estimate.
• Multiple drivers in the same queue report their position and speed, and the server concludes they refer to the same event and bundles them into the same message.

None of this is implemented yet – at the moment, the server will simply collect and forward messages. Implementing the above will be another milestone and will require some kind of algorithm capable of answering the following questions:

• How do we determine two messages from different sources refer to the same event? For example:
  • Locations overlap
  • Both messages refer to a stretch of road enclosed by two junctions (making it impossible to encounter the location of one message but not the other), so they probably refer to a situation which extends across both locations
• How do we deal with two related messages containing different information?
  • If the information is complementary (e.g. one message reports roadworks, the other a queue), combine it (the resulting message will report a queue due to roadworks)
  • If there are contradictions, figure out how to resolve them:
  • Detect bogus data (a driver stopping on the shoulder due to a vehicle defect is misidentified as being in a queue)
  • The situation may have changed over time (newer message wins)
  • Some parameters of the situation may vary within the location (traffic on a long affected stretch may flow at different speeds in different places, extrapolate and/or calculate an average value)

If we keep past data, this algorithm could be implemented into generate a forecast (e.g. traffic jams tend to happen Mondays to Fridays, during the morning and evening peak hours). Though I would note that this can get quite complex: we’d have to sort out anomalies (e.g. traffic accidents or roadworks slowing down traffic more than usual, public holidays, school holidays), as well as consider upcoming anomalies (e.g. scheduled roadworks or holiday schedules). If we get an actual report from a location for which we have a forecast, we need to figure out how to correlate them.

The TraFF transport format currently does not provide an explicit mechanism for forecasts, but that could be added. The important thing would be to clearly mark forecasts as such, so that legacy receivers will not mistake them for actual reports. Also, bandwidth will need to be considered, as only a small subset of all road stretches will have actual messages at any given time, thus a comprehensive set of forecasts for each and every stretch of road will likely be at least an order of magnitude larger. (Location references in TraFF are currently for a single stretch of road; there is no such thing as, say, “all national roads in Poland”.)

To my knowledge, the only existing source code for collecting events are the TraFF server as well as the Android apps, and none of these currently has capabilities to merge events or generate forecasts. In Navit I was kind of forced to do this (we ultimately have to match an event to the map and stick attributes to it). Here, events translate to “traffic distortions”, which can result in a lower speed (0 to indicate closure) and/or an extra delay for the affected segments. The (admittedly crude) solution is to assume the lowest speed and longest delay of any traffic distortion referring to a given road segment.

As for Java on the server, that was a quick win related to how the ecosystem evolved. The Android apps (Qz and RoadEagle) came first, and with RoadEagle came the conversion libraries – which are in Java. Implementing the server in Java allowed me to use the same conversion libraries for both the Android app and the server.

None of this is set in stone – if someone comes up with a better server implementation, I’m fine with that as long as the new implementation provides the same API. Server architecture could be reevaluated as well – it might make sense to have a core server which just manages messages and subscriptions, with third-party message retrieval/conversion being done by services external to the server (and written in a different programming language than the server) and fed into the server through a yet-to-be-defined API. (This API can be server implementation-specific, or, once we have an API for reporting user data, that could also be used for server-to-server communication.)

[mvglasow]

Any news on this? If you were to tell me TraFF and any ecosystem around it is still very much in its infancy, I would definitely agree. However, it’s sort of a chicken-and-egg problem: without a user base, few developers will come forward with contributions, there won’t be a workable solution for a long time, and until then, there is no user base and the circle closes. And the whole thing is kinda large for a single person to tackle. In order to get something workable off the ground, we’d need a few people joining forces and coming up with a minimum workable solution, so we can start growing a user base and attract more contributors. I’ll be happy to contribute, to the extent time permits. However, I will definitely need some help on extending Organic Maps to process traffic information.

[dikkechill]

I'd love to help, but have limited time at the moment.

To get this jump started I'd suggest to go for a PoC that allows people to use the basic functionality and then contribute to extend it in terms of areas, events and other functionality.

I suggest to limit the scope to:

• 1 country / area
• 1 or 2 traffic events (e.g. slow traffic)
• use traffic data from an official source (e.g. roadeagle to generate traff data for organicmaps, in other words no user contributed data)
• visualize data only (so exclude it for routing calculation for now)
• decide on an initial interface (maybe start with reading an XML file from the local file system first)

There are already some hooks from the Maps.me period (for data from a commercial provider) in https://github.com/organicmaps/organicmaps/tree/master/traffic

Maybe that part of the code is a good point to investigate:

1. Which traff data and events can already be mapped to the existing functionality
2. What needs to be done additionally (e.g. what must be fixed in the current and added as new functionality)

Does this help? Other thoughts are welcome

[mvglasow]

Great to hear that some functionality seems to exist already. How familiar are you with the code base? (Or who else can help with that part of the code?)

What I’d suggest for the PoC:

• Use one source (number of countries/areas then doesn’t matter).
• For traffic events, start with slow traffic/events and closures. This is what I also implemented for Navit and is probably the most straightforward thing to implement.
• For the source and interface, RoadEagle and subscription via Android intents and content providers would be a rather quick win. RoadEagle already has everything in place for that. And unlike a local XML file, we would get some real data right away. Only limitation: RoadEagle is available on Android, and a transport interface for iOS (or desktop Linux) has yet to be defined.
• Visualization as a first step (but I’d like to move on to routing in the next step – depends on what is already in place.)

[mvglasow]

Just looked around the code a bit and found two things:

• There seems to be OpenLR support (in the openlr folder), though I haven’t yet figured out where it is used. My first suspicion is that it might have to do with mapping the locations of traffic reports to the map (as the provider may have encoded the location using different map data, with slightly differing geometries). But that is just a guess from someone who is taking a first look at the code.
• The routing algorithm seems to be A (as suggested by file names in routing/base). There is an extension of A called LPA* (Lifelong Planning A*), which essentially adds the ability to change the cost of select edges and do a partial recalculation of the graph. We would need that to recalculate the route when the traffic situation changes – unless this is already implemented.

[biodranik]

A great start would be to:

1. Learn the existing code base and see how it works by providing some test/stub data.
2. Document it (what's there, what's missing, which data format is used from a server).

Then we can start with a server part and a client part in parallel.

[xephos1one]

Traffic data from https://github.com/bundesAPI/autobahn-api looks like to be by TomTom Intermediate Traffic. So the Data may only be allowed to be used inside the Autobahn App?

See https://mobilithek.info/offers/573189053963235328 And the linked details to the traffic data ("Verkehrsfluss") in the old MDM portal that got discontinued this year: https://web.archive.org/web/20231218115539/https://service.mdm-portal.de/mdm-portal-application/publDetail.do?publicationId=2841000 I saved that page end of last year as I didn't find it in the new Mobilothek and that still links to it. What we can see is that it's DATEX II Version 2.x using OpenLR as the Geo reference method.

Sadly I wasn't able to register to see contract information and valid from/to...

[zyphlar]

Some random thoughts:

I think it makes sense to contribute to something that's OSM-compatible in terms of licensing, as well as OM-compatible in terms of privacy and offline possibility

For example Google would be silly to route based on live data if there were only a few live data points. At some level it makes sense to modify routing based on average speed, not current speed. So we could be strategic and have a historical average data layer that either modifies estimated road speeds in the map files, or that is downloaded and parsed by the router separately

And then for live traffic incidents, rather than absolute speeds or slowdowns we could make routing logic simple by thinking of traffic as a % modifier to the road's mapped maxspeed or the measured average speed. Seems like routing based on single numbers is easiest

[auge8472]

@zyphlar The big task here is to provide the source of those data (both, historical average speed as well as current speed). Either OM (and maybe also other similar apps) feed a collector to build a data base or it is possible to soucre some of the databases that was mentioned above. Finally, data on traffic at different times on different days in different months is needed to improve the prediction of journey times.

At the moment, it looks to me as if the app calculates the journey time using only the permitted speeds and the distance travelled. My journey to and from work in a big city is 15km and the app estimates it to take about 24 minutes. Even at 2 a.m. with only green traffic lights, this is hardly possible, let alone at any other time of day. My real journey time during the day is between 45 and 65 minutes.

[biodranik]

The current inaccurate ETA is caused by missing penalties for turns and traffic lights. #1845

[auge8472]

The current inaccurate ETA is caused by missing penalties for turns and traffic lights. #1845

I have already surmised this as part of the problem in a comment on another issue that I can't find right now. 🙂

[Korb]

Google would be silly to route based on live data if there were only a few live data points.

Google has the largest array of data on both operational, current conditions, and hourly statistics by day of the week for most road sections. Even in countries where the company is not represented. Thus, for long routes, Google can plan a route not only based on the current traffic situation, but also based on the expected traffic load. The latter is very important for routes longer than an hour.

The very first link in Google leads to the Traffic Layer | Maps JavaScript API. Could this be used for route planning?

Will this make the app fully "organic"? Will this improvement not cause the app to be flagged in F-Droid as containing things "you might not like"?

Also, I would like to point out that in countries where the majority of the population speaks Russian, Yandex' maps are very common — JavaScript API Reference / traffic.provider.Actual. Honestly, I don’t even know what’s worse for paranoids: a global corporation or a local company run by bandits, whose user data is freely available on the darknet?

I would like to point out separately that drivers need traffic information not only for automatic route planning, but also for coloring roads in car-navigation mode.