search

opentraffic / architecture

OTv1 overview
71 stars 11 forks source link

Using OSM GPX planet as a data-source #32

Open laurentg opened 9 years ago

laurentg commented 9 years ago

Announcing the GPX Planet. 2.6 Trillion GPX points.

It could be nice to be able to use this (or part of it) as a GPS source.

kpwebb commented 9 years ago

Let's ingest it! Just need to get the traces into a time sorted format with and we're good! We just need:

trace_id,time,lat,lon

2.6 trillion points will take a bit of crunching but we're currently getting about 15k points/sec/thread (and lots of room still for optimization) so that's 2000 days on a single thread. Scaling up is linear by throwing more CPU power at this.

laurentg commented 9 years ago

There exists regional extracts that can simplify things where only a small region is needed (GPX files in each extracts are stored by lat/lon bins). Also lots of those files are walk / bike / boat / planes etc... traces, a pre-filtering by mode would help reduce the amount of data to process (altough some heuristics are needed as the traces do not have mode information attached to them).