search

opentraffic / reporter

OTv2: distributed service that matches raw GPS probe data to OSMLR segments and sends anonymized speeds to Datastore
GNU Lesser General Public License v3.0
13 stars 16 forks source link

allow the reporter to work on a probes traces one point at a time #38

Closed kevinkreiser closed 7 years ago

kevinkreiser commented 7 years ago

this removes the requirement that requests have to be more than one point. this should allow for a streaming architecture such as kafka to sit in front of the reporter and pump in single points. this puts considerably more burden on redis as it has to keep in memory possibly a bunch more trace data.

the running characteristics of this change seemed ok at first but there is a considerable issue that i think we should tackle before merging this. ill note it in the diff

dnesbitt61 commented 7 years ago

Can you replace the last point in the trace with the new point if the distances are below some threshold? That way the trace size doesn't grow when the vehicle is stationary? Or maybe replace the time but not the position (so a case where vehicle movement lower than the threshold doesn't continually drop points).

kevinkreiser commented 7 years ago

@dnesbitt61 we should use the same threshold the matcher uses for skipping points.

im also thinking we dont want to worry about time for the sake of accumulation in the reporter because the same problem will occur.

roughly 75% of all the matches run are due to this problem so we really need to reduce this number.

kevinkreiser commented 7 years ago

ok ive attempted to mitigate the standing still issue by using distance and elapsed time as a gauge of when we've accumulated enough to warrant matching. that didnt really bring down the amount re-matching that ended up being done.

the only way to fix it completely is to do the other suggestion i had which was inserting invalid segments to signify when the trace leaves edges with segments to go into edges without segments. in this way we can know that the last thing the person did wasnt actually on a partially traversed segment and can drop all of that shape. ill make an issue in valhalla to track that.

additionally i stumbled across a problem that was there before in the reporter but is exacerbated by allowing single point trace updates (due to threading). ill comment on the code in line

kdiluca commented 7 years ago

thanks @kevin