API data is flawed at best, misleading at worst

[UmbyUmbreon]

Before I start, I'll preface by saying that I use HERE Maps on a near daily basis for enterprise development, so I have a fairly decent working knowledge of the platform.

This project is using a flawed data philosophy to try and track speeds on certain roads or sections of roads, and the use of "a vehicle was detected" in tweet text is tricking a lot of people into thinking that this is tracking individual speeding drivers, when this is not the case.

HERE Maps' traffic flow API is not tracking individual drivers speeds down certain roads, but rather appears to be doing some kind of aggregation to determine the speed that traffic can flow on a small section of a particular road, regardless of the speed limit. The speed values aren't very well documented on HERE Maps, so I can't claim this is the case with 100% certainty, but regardless it's essentially the same type of system that turns road sections from green to red in the traffic layer on Google Maps, and not a means of tracking the speeds that drivers are actually traveling down a section of road.

Worse, this project is doing this without taking HERE Maps' confidence rating (CN) into consideration, where > 0.7 && <= 1.0 is (close to) real-time data, > 0.5 && <= 0.7 is historical data over an undetermined period of time, and <= 0.5 is whatever HERE Maps thinks the speed limit is on that road section. So in most cases, this is tweeting a speed that HERE Maps pulled out of thin air which it thinks you can travel right now, and in the worst case it's tweeting whatever HERE Maps thinks the speed limit is.

According to HERE Maps:

To only report speeds where real time data is available, we recommend that you include speeds with a confidence greater than 0.7 CN and include road closures with a value = -1.0 CN

[BerkshireCar]

Thank you @UmbyUmbreon it was never the intention to mislead - we have posted a pined tweet on twitter and will research further. if you or anyone else have recommendations on how to update the flows please do raise a pull request.

[BerkshireCar]

@UmbyUmbreon I've updated the 4 blueprints to reflect these comments and add in filters for realtime, thanks again for taking the time to document this and share. Please let me know if you feel more changes should be made before I close this issue.

[BerkshireCar]

Further research documentation and background

Ref: Here Traffic API Ref: StackOverflow

The XML schema has descriptions of the field acronyms, which is proving to be somewhat useful: https://traffic.ls.hereapi.com/traffic/6.0/xsd/flow3.1.xsd?apiKey=

_

Speed (based on UNITS) not capped by speed limit _ _ The free flow speed on this stretch of road. _ - Filters have been added on the CN object to only include real time data. - Twitter Output uses SU (Uncapped Speed) object not FF (FreeFlow)

[JHolah]

I'm still looking into this myself. I would love to be able to provide a plain English explanation to anyone who questions the data from my local speederbot.

Assuming we've updated filters to include only real time data and uncapped speed, is it fair to say that the output is based on realtime GPS data from within that last x minutes?

[BerkshireCar]

I'm still looking into this myself. I would love to be able to provide a plain English explanation to anyone who questions the data from my local speederbot.

Assuming we've updated filters to include only real time data and uncapped speed, is it fair to say that the output is based on realtime GPS data from within that last x minutes?

I'm trying to raised a paid support ticket with HERE developer support to confirm. Will post here if/when received.

[BerkshireCar]

After talking with HERE support on slack, to question the SU data validity and using this dataset below for context.

They have confirmed that it is safe to assume that at least one vehicle was traveling at the SU uncapped speed. In this example I specifically questioned the 43.83mph

This also assumes that the CN filtering is enabled to ensure Realtime data.

[UmbyUmbreon]

SGTM and LGTM! Good to finally get some clarity from HERE on what exactly the speed measures mean. Would be nice if this was actually covered in their documentation!

[BerkshireCar]

Here is a summary of the information I received from support regarding the source of SU data for reference.

The uncapped speed (SU) is derived from data gathered along that particular section of roadway. This could be probe data from vehicles and/or sensor data, it's not just taken from a single vehicle, but rather multiple over a period of time. It is safe to say that there was at least 1 vehicle travelling at the uncapped speed.

• Probe data is from vehicles.
• Sensor Data means from fixed sensors built into roads that detect traffic moving across it.

[jeggburt]

"Sensor Data means from fixed sensors built into roads that detect traffic moving across it."

That's quite interesting; I'm wondering if this includes the sensors that are linked up to traffic signal junctions, or if they are specific sensors independent from traffic signal junctions, or a hybrid.

[JHolah]

This is very interesting. I would assume that the sensors built into the road would not include traffic signal detectors, as these are induction loops that check for the presence of a lump of metal and I don't think detect or record speed. I would have thought this means those temporary pneumatic cables with a box at the side of the road, often used for surveys, that can detect speed and direction.

It is great to get proper confirmation from HERE devs. Equally disheartening to see that these are AVERAGE speeds of AT LEAST ONE vehicle. I have updated my tweet template to reflect this. But then, exposing this road danger is exactly what the project is about, right? Thanks again for your efforts.

[jeggburt]

It's entirely possible there are induction loops in the ground, similar to those used for traffic signal junctions.

I've done an liitle bit of investigating for the roads around the Tyne Tunnel.

A187

A19

[A193] (https://www.google.com/maps/@54.9991095,-1.488049,3a,75y,66.57h,52.68t/data=!3m6!1e1!3m4!1sLoz-Y3ITImNvR3ji7HZqTA!2e0!7i16384!8i8192)

These are two-way loops and there is a small box close to these as well.

If you look at the Google Maps links, you'll see that it's implausible that these are being used for traffic signals, due to the lack of traffic signals within fixed points of the exact locations.

It's knowing roughly how close the output relates to the area you're looking at, and where to find them, for accurate reporting.

In the absence of HERE confirmation, I think it's a reasonable ascertion to make.

[JHolah]

In that case, I think you're right. I don't recall ever noticing them in that situation but your example makes perfect sense. In the area I'm monitoring, I only see the ones at traffic signals, but then it is an entirely different scenario (20mph residential area).

[BerkshireCar]

HERE API Developer support could not be more specific about the source of the data other than the details in my post above I'm afraid. The project was simply setup to highlight concerns with speeding hope it's useful to others.

[JHolah]

That's fine. Explaining that at least one vehicle was travelling at that speed is sufficient. 👍

[BerkshireCar]

Support have just confirmed "Probe Data" could be sourced from Apple and Google phone GPS data. Hope this helps.