search

codeforanchorage / realtime-bus-sms

Text back current arrival time for buses given a texted in bus stop number for People Mover. AddressLookup branch would allow texting "Arctic and Fireweed"
https://bus.codeforanchorage.org
14 stars 10 forks source link

Improve file-logs #123

Open mark-meyer opened 7 years ago

mark-meyer commented 7 years ago

The bus app currently uses lowDB to write JSON file-based logs to disk. The way lowDB works requires it to hold the entire log in memory and then overwrite the entire log file with each hit. The current db.json file is 62MB and about 2.7 million lines! That's a lot of overhead for each hit and since there are two log files the app write both with each hit.

I propose moving to something like neB https://github.com/louischatriot/nedb (or similar if anyone has another idea).

The main advantages neBD has are:

  1. It writes by appending to the file rather than rewriting the file.
  2. It writes one record per line, which will make it easier to rollover old logs by simply truncating them.

The interface is about as simple as lowDB. It still hold the db in memory, which is less than ideal, but manageable if we can automate log rollovers.

It also has indexing which might speed up the /logPlot which is currently slow (and I think it also blocks).