Possibility of avoiding traffic jam?

[wehkah]

I don't know if it's possible for Osmand, but it would be an enormous relief, if it was possible to access traffic information databases, that include traffic jam, construction sites, road closings, etc, and include these information when calculation the routing.

[OpenSourceForEver]

https://github.com/osmandapp/Osmand/issues/1432 was closed because there is no possible traffic source. Does anyone know any possible data source?

[mahdi1234]

@vshcherb @OpenSourceForEver I guess it might be country-by-country e.g. Czech Republic has national registry which can be used freely after signing agreement (I suggested this source to other app which already works on it)

http://registr.dopravniinfo.cz/en/index.html http://registr.dopravniinfo.cz/en/sources/cz-ndic_ddr-common/

[wehkah]

I've seen apps in the PlayStore that claim to be able to display traffic jam in realtime. I haven't tested them yet, but I guess there have to be some databases that theses apps retrieve the information from.

Most FM radio stations also offer web pages that provide traffic information.

So, I would guess, these information are neither top secret nor anyone's private knowledge. After all: these information can be seen by anyone who is looking at the corresponding streets, so how could those information be private property? Ans on the contrary, the spreading of these information is in everybody's interest, isn't it?

[exploide]

So, I would guess, these information are neither top secret nor anyone's private knowledge. After all: these information can be seen by anyone who is looking at the corresponding streets, so how could those information be private property? Ans on the contrary, the spreading of these information is in everybody's interest, isn't it?

Ideologically you are right that this information should not be private property. However, that's not how it is as of today. Google for example collects its own traffic information utilizing the mass of Android phones with GMaps installed out there. This gives them the advantage over competitors like Osmand to have great traffic information available. Hence, they don't share it.

"Public" information from FM station's websites on the other hand is rather inaccurate, not perfectly machine readable (i.e. not necessarily structured) and lacks sufficient coverage.

So what the world (i.e. we :stuck_out_tongue: ) needs is an open data source for traffic information where everyone can participate. But as stated in the other issue, the OpenTraffic initiative did not succeed (yet?!).

It looks like Mapbox offers a traffic data service now that's usable for OSM-based apps. https://www.mapbox.com/traffic-data/ Unfortunately, I doubt that this is open data. One needs to pay for access. Don't know how exactly this works, but maybe Osmand could use that source for free when they participate in data collection.

[mbirth]

Maybe support for TMC would be a starting point? With a cheap RTL_SDR device, you can receive and decode it easily.

[scaidermern]

How about starting to collect this data from OsmAnd users, possibly in cooperation with MAPS.ME and similar OSM apps? Optionally, of course. Every user can choose whether to participate or not.

This will of course generate a lot of data to process server-side. However it looks like this could be a chance to finally get live traffic information as open data.

[mvglasow]

I have been working on something like that over the past years. At the core, there is a generic exchange format, with a logical data model loosely inspired by TMC, and locations expanded to WGS84 coordinates. More information on the exchange format is available at http://traffxml.org.

Data is provided by external apps; consumers (Osmand in this case) only need to add the capability to receive and parse TraFF data and translate it into their own data model. As there is a standardized exchange format, there is no need to update the consumer app when new sources become available. I can provide some first-hand experience on processing TraFF data on the data consumer side. There is also a Java library for the TraFF object model, which I could expand to parse TraFF data in XML format.

The abstraction layer makes the ecosystem of traffic data sources independent from the apps used, thus multiple apps can use the same ecosystem.

As of now, there is only one live data source: Qz, an Android app, which receives TMC data (requiring an external FM/RDS tuner), converts them to TraFF and forwards them to consumers. See the project home page for more information; it is available on F-Droid. Another one, Roadeagle, is currently being developed. It retrieves traffic information from open data portals and provides TraFF data to applications. It currently has support for Poland and Lithuania (both of which lack free-to-air TMC services). At the moment I am working on Datex-2 support, which would open up a range of new potential data sources.

Future items on the roadmap (contributions VERY welcome):

• Review the TraFF transport mechanism: Currently it is specified only for Android, and the current spec needs improvement. We will need to expand that to other platforms (Osmand will likely require iOS support).
• Extend the TraFF specification with a subscription mechanism (limited to certain geographic areas and/or events) and differential updates (returning only what has changed since the last feed was delivered to the subscriber)
• Set up a cloud service which collects data from all known sources (including those which are free but require registration) and provides them in TraFF format
• Give that service a certain level of “intelligence” to deal with multiple reports affecting the same location (or overlapping locations)
• Enable that service to accept user submissions (this mainly requires a certain level of protection against malicious submissions)

@mahdi1234 The Czech sources require the recipient to register (free of charge) and run a server—this would definitely be a candidate when we get to the cloud service phase, but would not be feasible without.

@mbirth Re SDR, decoding RDS-TMC is very processor-intensive due to the high sample rate required. I got it to work on a laptop but found it beyond the capabilities of a smartphone CPU. However, external FM tuners (such as the very widespread Si4703 and the GNS FM9-BT) work well.

[mbirth]

@mvglasow The idea with the SDR receiver was more like volunteers decoding RDS on something like a Raspberry Pi (or their PC) and uploading it to some central server where the data is then distributed to OsmAnd users.

Although I've learned that TMC and TMCpro have an extremely slow reaction time and the data might only be useful if you have absolutely no other means of receiving more up-to-date data.

[mvglasow]

Not sure about the legality of re-sharing decoded TMC data, but if we can sort that out, it is surely worth looking into. Otherwise, Qz with a local dongle has served me well for about a year now.

Reaction time is essentially up to the provider and might differ between one and the other. In some places we have a choice, in others we have only one or the others. See https://gitlab.com/traffxml/roadeagle/wikis/Data-Sources for countries where I have found open data portals, and https://gitlab.com/mvglasow/qz/wikis/countries for countries where TMC with Qz works.

[Zireael07]

MAPS.ME does have a traffic overlay, although I am not sure where the data comes from.

[mvglasow]

Apparently they are not disclosing that, see here: https://github.com/mapsme/omim/issues/5122#issuecomment-276597095

Judging from further comments on that issue, the required information is pulled in from a non-public repo—which means even analyzing the source code will not give us that information, and non-official builds will not include that traffic information.

On the other hand, it seems they have the infrastructure for handling traffic information in place. Which means one might be able to fork the project and connect different traffic sources to it. One would have to analyze the code to find out what format the traffic data would need to be in (it seems they have support for OpenLR for location reference), and how/where to provide the traffic data. That part is probably off-topic for Osmand, except that one could possibly borrow some ideas from the code.

Then again, I have implemented traffic processing in Navit and am happy to provide implementation advice.

[wehkah]

Any progress on this issue?

The other day I had an idea:

When you are stuck in a traffic jam, there are quite a lot of vehicles around you, most of them for nearly all the time the jam lasts. In 90% of every vehicle today, there's some kind of active bluetooth device: a radio, a head set, or just another mobile phone.

An Osmand plugin could, if the user permits it, memorise all these unknown blutooth devices around and if this list doesn't change for a certain amount of time while your own speed is very low or even zero, it's very likely that these devices and yourself are stuck in a traffic jam. This information could then be sent to a server, from which other Osmand users could download it, using the same plugin.

What about it?

[pebogufi]

The range of bluetooth will not be much more than about 10m (on most smartphones) so i suppose it would not work. And there will be a problem with laws concerning privacy.

[scaidermern]

Besides consuming additional battery this is not really necessary to determine if there is a traffic jam. OsmAnd just needs to compare the current speed against the maximum allowed speed, that's all.

Apart from that it is also useful to store the typical average speed for a particular road and a particular time to improve route planning and predict traffic jams. However this is obviously more advanced and could be done in a later step.

[wehkah]

10m will cover at least two vehicles on every side, that would be 5~10 vehicles.

About law issue: I don't think it's illegal to memorize the MAC addresses of surrounding devices, Android does this anyway when you activate BT. But the received MACs could be hashed before uploading to a server.

But scaidermern is absolutely right: comparing the speed to the speed limit would also be sufficient. This data can easily be rendered anonymous before uploading, and later it can be used for all kind predictions. For example: if a certain amount of people do not follow the calculated route on a certain segment of a street, but drive around it, you can assume there is a detour/construction site. This information could than be used to suggest alternative routes.

[yareckon]

OK, this got way general, when a specific implementation suggestion was already dropped into the thread. @mvglasow provided a great way to fix this for some users, i.e. a parser of TraFFXML that is already a java library, and is already integrated into Navit. Whether folks get TraFFXML over the internet, or via a FM radio decoding app like QZ, that option would be really awesome. Could someone take a look at this? If it's already a Java library, it likely would be more of an internal architechture issue in OSMAND about how to overlay the traffic cost data, right?

[Zireael07]

Just comparing the speed to the speed limit would probably be enough.

[mvglasow]

By now things have progressed a little further. I am about to finish version 0.8 of the spec, which introduces proper subscription management (so consumers like OsmAnd can tell the source which data, e.g. pertaining to which part of the world, they are interested in). Version 0.8 also introduces HTTP[S] transport.

Code can be found at https://gitlab.com/traffxml. Among others, there is:

• a TraFF parsing library (which reads TraFF XML and converts it into Java classes, which can then be processed further)
• a TraFF transport and consumer library for Android (there is none for HTTP, but that is much easier to implement by hand than Android)
• a TraFF server: a combination of daemon and Java servlet which aggregates data from various sources and publishes them as TraFF
• conversion libraries for format used by various external sources—currently mostly focused on Europe
• RoadEagle, an Android app which retrieves data from various external sources and exposes them in TraFF format (unfortunately, the Datex-2 converter is not compatible with Android, hence RoadEagle does not support Datex-2 sources; these can only be made available through TraFF Server).

For some things you might still need to look into the dev branch of the respective repo, as TraFF 0.8 is not completely finished yet.

@Zireael07 One challenge is to match the message to the road segments. TraFF has a coordinate pair for the start and the end of the affected segment (plus one auxiliary one); these can be off the road, depending on the source of the data—they use all kinds of map data for their reference, not just OSM. Additionally there are some attributes such as road ref and road class, which can be used to aid matching. The cleanest way is probably to route from one point to the other (which is how I implemented it in Navit): the cost is mainly determined by segment length, with a certain penalty factor: an off-road link from/to a reference point is the most expensive; a road link gets cheaper the better the attributes match.

The other is then to do dynamic rerouting: if a new traffic message comes in while a route is active, we want to recalculate not the whole route (which would also work but be inefficient) but just the part that changed.

[kroeckx]

So is there a possibility to to use a Datex2 as source? At least some European governments seem to make such a data source freely available. There is also data available like parking availability.

[mvglasow]

Datex2 data sources are available for many parts of Europe, though the events covered vary: some have live traffic information, others only have relatively static information such as roadworks. The main challenge is that the XML schema is complex: deserialization pretty much depends on being able to auto-generate Java code, which results in ~200 classes.

I have tried a few XML deserialization libraries, and JAXB is the only one that I was able to get to work and which is capable of auto-generating Java classes from an XSD. Unfortunately, it is not available on Android and also cannot be imported, as it uses the reserved javax namespace.

The other conversion libraries rely on the Simple XML Framework, which works on Android but does not provide a native way to generate Java classes from XSD. There is a third-party tool which claims to do that, though I haven’t tried that yet.

As for parking availability, we would need to define the relevant events in TraFF first, then this information could be used as well. Osmand would still need to process that information, which is likely to be very different from routing around traffic jams. But it surely is an outlook for the future.

[gituser789]

May this is useful (available for germany): Since a few weeks there is a new app available provided by the german highway company. I think some volunteers do some reverse engineering to find out how the api is working. https://github.com/bundesAPI/autobahn-api Example for warning data structure (depends on the actual warnings, when you open the link) https://verkehr.autobahn.de/o/autobahn/A7/services/warning

[mvglasow]

Sounds interesting. We would need to figure out the terms under which the API is available and the data can be used. If they are under a free and open license, I might integrate that into RoadEagle. (So far the Germans have only provided this kind of information in Datex-2 format. I have a working Java parser/converter for that, but it is not compatible with Android and would be a huge effort to port.)

[mvglasow]

seeing this article (https://www.heise.de/news/Nachhilfe-fuer-den-Bund-Freiwillige-dokumentieren-Programmierschnittstellen-6158153.html), it doesn’t look like the API and the data retrieved through it are truly free and open, at least for the moment. Hopefully that is going to change in the near future...

[budda85]

Maybe this would be a possibility to include traffic jams in OSM. But this is only available for Germany. https://autobahn.api.bund.dev/

Edit: Sorry, i have not read the last posts!

[KlavsKlavsen]

Google maps has ONE big benefit over osmand - which is that it forcefully steals every Android/maps users location data - and misuses for commercials - while also enabling to know where there's traffic jams, slowing speed.

I believe this could be reached by an Open Source solution - which hopefully could then be adapted by ANY routing application (and hopefully EU etc. would consider disallowing the current Google approach - which AFAIK actually IS a breach of the GDPR).

While hoping for the law to catch up to the situation, I would really like if a proper solution was put in place to show it SHOULD be done.

I'm thinking osmand should time "a segment" of travel (between signals or turns f.ex.) - and compare to its "calculated cost" - this can be done offline.. and then it could simply report something identifying the route-segment (the route-segments should be split in such a way so all users travelling same stretch of road will calculate same segment) - and the time it actually took and a percentage of how much "off" from expected time, it was (or only the percentage).

This WOULD require some service running to store it - and I'm hoping openstreetmaps would see the use for such a service - as that would enable ANY client to pull data from openstreetmap about "expected variation" - which could simply conclude that from the "timed route-segments" it got - for each segment.. So client would submit a list of "route-segments" in the route its calculating - and get back the "expected variations in percent - if any". These variations could be stored as simple files - so the clients could simply "figure out the path to GET" (which would mean timerange is part of path) - and thus data could be served as simple files (cheapest performance wise).

This would also help detect anomalies in osmand route calculations and could possibly be used to detect other anomalies - without compromising anonymity.

[breversa]

Though not open-source (yet ?), Magic Earth (https://www.magicearth.com) both uses OpenStreetMaps AND has traffic warning. I don't know how, however, it's the navigation app by default in /e/ OS (https://e.foundation/) and I believe there are talks between the two to open-source the app. Might be worth having a look ?

[hermann-san]

someone mentioned above that Mapbox is a possibility. They are offering/selling it to external customers. In turn they ask for anonymous traffic data from the users. Shouldn't be a privacy issue then if OSMand can anonymously send that date to mapbox.

One pricing example 5 Million Requests / Month would be 5.000$ / month This could be sold as an OSMand plug-in. I would be ready to regularly pay (e.g. yearly) for that plug-in. Navigation apps like Copilot also sell traffic data to the user as a plug-in for a yearly subscription.

maps.adac.de also has traffic data. Not sure from where.

[gituser789]

https://www.verkehr.nrw/ also includes traffic data. Can not find any hints from where. Background map is OSM.

[scaidermern]

someone mentioned above that Mapbox is a possibility. They are offering/selling it to external customers. In turn they ask for anonymous traffic data from the users. Shouldn't be a privacy issue then if OSMand can anonymously send that date to mapbox.

One pricing example 5 Million Requests / Month would be 5.000$ / month This could be sold as an OSMand plug-in. I would be ready to regularly pay (e.g. yearly) for that plug-in. Navigation apps like Copilot also sell traffic data to the user as a plug-in for a yearly subscription.

How is the coverage of MapBox? Some while ago I heard they have good traffic data for the US but anywhere else it is of no use. Has this changed?

[hermann-san]

@scaidermern they say they have coverage for western europe . They also say that they can integrate with here.com or TomTom. So if they already use those data points, then the coverage should be at least usable. If not, they why not helping to build up a usable base? Since the usage is billed on the number of requests, it's shouldn't be too expensive and if it's not yet usable, I guess one can make a deal with mapbox and only pay the fee's once it's usable for a specific country. Meanwhile one can still provide traffic data to them. If Magic Maps also use Mapbox traffic data as a data source, the it's fine. I've used Magic Maps before. Surely, if someone from OSMand will contact Mapbox, they will tell for which country there is sufficient coverage. I would just go ahead and contact them to see what the say.

@gituser789 good info. Maybe they will provide (sell) the data it to external applications . This what they say about their data sources. ie o.g. Informationen werden bereitgestellt von: Verkehrszentrale NRW, Landesbetrieb Straßenbau NRW, Polizei, © TomTom, Ruhrpilot, Stadt Düsseldorf, Mobil-Im-Rheinland. It's for Germany , Belgium, Netherland, Luxembourg I guess one also simply needs to talk to them.

[Dirk--]

[...] I guess one also simply needs to talk to them.

just asked (via email) kontakt.verkehr@... to join this conversation ...

[vshcherb]

OsmAnd already has custom Online Routing support, so it could be used for testing purposes

[Lee-Carre]

List of publically available data for current road conditions, though for Yank-land, most of those are (going to be) the per-state sites of / for the Open511 traffic info system.

[hermann-san]

Hey @Dirk-- have you received any feedback from the NRW people?

BTW. it is possible in OSMand to add a Online Map Source with Google maps as the source. It won't work with route data, but traffic info can be displayed on the map. This is the URL http://mts0.googleapis.com/vt/lyrs=m,traffic&x=%7B1%7D&y=%7B2%7D&z=%7B0%7D Of course this is not a proper replacement for "native" support of traffic data in OSMand, but at times it may help and I would only turn this on when I desperately need it. Else it would be disabled. Similarly one could simply add a browser shortcut to the homescreen with a link to Google Maps (in the browser) or which ever map webpage with traffic data there is. As I said, this is just a workaround until traffic data is available in OSMand :-)

[Dirk--]

@hermann-san

have you received any feedback from the NRW people?

no reaction at all :-(

[hermann-san]

this issue has a "nice to have" tag.
For me personally this is an essential , important feature when I'm often driving routes with the car that have traffic jams. This may be the case for many more people who drive to work daily, needs it for business trips,, for shopping, in holiday, etc..

[Raphty]

I so much agree. I purchased OsmAnd live because I thought it would get live traffic data (updated once per hour) but well...

I am not mad, because I do like the app and what they do... but without traffic data I will always opt for Apple Maps, because the traffic is the only reason I use maps on a daily basis. I know the routs, I just want to know if I should take another one due to high traffic.

So for me, also considered essential. And I would be willing to pay extra.

[brunofin]

A quick search for "open traffic map" gives me this as a first result:

https://opentransportmap.info/

It's for the whole UE and it's based on OSM. I know it's not a world wide source but it's a good start.

There are similar results for "osm traffic".

Regarding traffic source, similarly to how osmand has a UI to pick different routing providers, it could as well implement a UI to pick different traffic providers.

[scaidermern]

A quick search for "open traffic map" gives me this as a first result:

https://opentransportmap.info/

It's for the whole UE and it's based on OSM. I know it's not a world wide source but it's a good start.

Well, the description says:

• visualizes average daily Traffic Volumes for the whole EU
• visualizes daytime related Traffic Volumes in OTN Pilot Cities (Antwerp, Birmingham, Issy-le-Moulineaux, Liberec region)

• Average daily traffic volumes don't help at all at avoiding traffic jams.
• Daytime-related traffic volumes are better than nothing if no live data is available. However, they exist only for very few regions.

[brunofin]

I think probably at this point OsmAnd would benefit from collecting it's own traffic data, but the idea here is since this is such an important public information, I believe this should be a different project fully decoupled from OsmAnd, but in which the OsmAnd app could have its own official plugin.

So here's my idea:

• A standalone open source and public service built on already existing open standards for representing traffic data (if those do not exist yet, this is also a good opportunity to create such);
• The service should not be built upon OSM, but rather be compatible, so we don't accidentally create a "brand lock" (if the service succeeds, it would be great if the official OSM would display it as its own layer nonetheless). We could also try to get the people from OSM involved;
• Standalone apps dedicated to collect traffic data. This is so even if users pick to use a different navigation app, they still can chose to contribute with collecting data. User should have full control on the granularity level of info they want to provide, it should be fully transparent for the user what info is being collected, where is being sent to, and how it's going to be processed, and for how long it is going to be held. Info can be anonymous or not;
• Simple and well documented API so other navigation services can write their own plugins and integrate with it. The idea is that if you use a navigation app with an integration plugin, the standalone app is not needed;
• Finally, user manually reporting things such as traffic, accident, speed traps and police (where legal), road closure and events, all those kind of real time events that aren't just live/historical traffic but also affects ETA and routing itself (such as Waze) would be an essential goal to aim for but probably best to keep it as a nice to have for the beginning.

If this kind of service and standards do not exist, I think this is a big deal, probably if this is well though and well made, governments will get interested in integrating their own data with it the same way as most governments also automatically update OSM with their own official maps.

I would be eager to collaborate with such projects, so definitely would like to help if the tech stack matches the ones I am familiar with too (mostly the JS techs).

[dumblob]

@hermann-san suggested collaboration with Organic Maps in https://github.com/organicmaps/organicmaps/issues/1160 .

WDYT?

[hermann-san]

@dumblob I don't quite understand your question. Are you asking how to collaborate with Organic Maps? Maybe like brunofin has suggested above. I don't have further idea's but I guess if one reads the above messages there should be some obvious hints. E.g. the dev's could also simply exchanging idea's and experiences. test2

[dumblob]

What is the first step then and who will do it?

[hermann-san]

I would say the first step is to get a statement from the OSMand product manager or lead developer about the current state of this idea or project. If they are not willing to cooperate, then there is no point to continue with the task. Also there may be things already in the works and we just don't know it. I can't remember that I've ever seen a statement from OSMand people on this idea. Maybe I've just missed it, but I can't find it in this GIt issue. I'm assuming that they are reading this issue and nobody has to specifically contact them, but maybe I'm wrong ;-) and someone has to directly ping them... maybe brunofin should contact the OSMand people and kindly give us some feedback. brunofin has some good idea's and wants to contribute something to the project

[dumblob]

Ok, just to be sure - ping @Chumva @vshcherb @cepprice @alex-osm @ivanPyrohivskyi @nazar-kutz @xmd5a2 ...

[derkrasseleo]

I think an opt-in feature for tracking users speed could be a huge improvement to Osmands routing capabilities. Trend + current activity = prediction, right? Could this temporarily tested by developing an extension for Osmand? I know this feature is going to take a long time to develop but for me personally, the lack of traffic data and reviews in Osmand keeps me from using it, at least for routing.

[scaidermern]

I think an opt-in feature for tracking users speed could be a huge improvement to Osmands routing capabilities. Trend + current activity = prediction, right? Could this temporarily tested by developing an extension for Osmand?

This feature is way to complex to implement it just for testing purposes. Tracking of speeds is the easy part. Processing this data is probably way more complex.

[hermann-san]

A new project that may help with developing a live traffic data system - The Linux Foundation -> Overture Maps Foundation https://github.com/organicmaps/organicmaps/discussions/4102

[scaidermern]

A new project that may help with developing a live traffic data system - The Linux Foundation -> Overture Maps Foundation organicmaps/organicmaps#4102

I strongly doubt that these big tech companies will make their live traffic data available for free or even sell them for a reasonable price. But let's see.

[dumblob]

I strongly doubt that these big tech companies will make their live traffic data available for free or even sell them for a reasonable price. But let's see.

Even then it should be beneficial to talk to them and maintain an active relationship. Perhaps some API alignment etc. could make it easier & faster for (other) data providers and OsmAnd, Organic Maps, ... to cooperate.