Reorganize roads into segments

This will be a rather complex task. Basically, we'll need to do something similar to building a routable database, where we normalize the OSM roads to our needs. This includes:

Joining ways that are the same road and are split in OSM to attach different tags.
Split ways on junctions.
Split ways by length. It doens't make sense to compare events on a segment longer than 100m or so, maybe 200. After that, the events on both ends really don't have much to do with each other.

Maybe in a later stage we can go even crazier:

Normalizing twoway roads to oneway roads. They are duplicated, one of them is reversed, and both are shifted by half the road width to the driving side (right for right hand traffic). Also normalize oneway=reverse tags by reversing the geometry. Then we can treat all ways in the database as oneway and directional.
Remove mini segments on big junctions that are used to represent turning lanes etc, as bicyclists probably do not use these, and we cannot accurately attach to them. Instead, we could generate a circle (aka point + radius) "Junction" entity and assign events to that instead of the road, then do statistics based on the driving direction or (incoming, outgoing) ways pair used.

I'm not sure which kind of technology could be used for this. Building this directly in osm2pgsql sounds unreasonably complicated, I don't think Lua has the libraries for that. I'd love to be proven wrong though ;)

The naive approach would import a lot of stuff into memory, then iterate over ways and try to fix them. This sounds like it would quickly go O(n^2) or worse though. It might make sense to do some smart spatial organization for this. It probably also needs a few passes over the dataset, similar to what I describe above, and storing in GeoJSON in between is probably going to be huge in memory and very slow. Remember, we'll want to do this with e.g. the whole highways network of Germany, which is ~3GB of spatial data in postgis (I think).

So the details of this are not clear to me yet, but we should do it sometime soon so we can have proper analysis on road segments that make sense for our application.

openbikesensor / portal

Reorganize roads into segments #115