search

opentraffic / reporter

OTv2: distributed service that matches raw GPS probe data to OSMLR segments and sends anonymized speeds to Datastore
GNU Lesser General Public License v3.0
13 stars 16 forks source link

use less inodes and readme for simple reporter #113

Closed kevinkreiser closed 7 years ago

kevinkreiser commented 7 years ago

this has updates for the readme but also updates to use less inodes and has a hard limit on the number of file the process can create.

this runs in appx 6m 20s, which is more than a minute faster than the previous version.

kevinkreiser commented 7 years ago

i just tried this without using tmpfs and its just as fast 6m22s. the max number of files it makes per run is 64k

kevinkreiser commented 7 years ago

i actually knocked it down to 4096 total files and that still seemed to perform in the 6m15sec range so i think we have reached the lower limit with this approach. i think this is good because even if a days worth of data is 50gb that average file will only be 15mb which is fine for a thread to load into ram